Tag Archives: KB

VMware – vSphere Replication 5.8 and Custom Certificates

Posted on by

I waffled on some time ago about using proper certificates in your vSphere 5.5 environment. You can read about some of how to do that here. Eric has a nice summary of the steps here. I got a call recently from the customer about a few things and they mentioned some issues with vSphere Replication 5.8. Turns out I’d forgotten about vSphere Replication when I’d gone through the certificate replacement process, as it was done as a PoC. The fix is simple: power off the appliance and power it on again. VMware has a KB for most every situation, including this one – VMware vSphere Replication appliance no longer able to communicate with the VMware vCenter Server after changing the vCenter certificates (2063955). It also helps that I’m a bit late to this particular party.

The next step should be to replace the certificates on your vSphere Replication infrastructure as well. I was going to put together a post on that too, but it’s probably simplest if you read the VMware KB – Configuring CA Signed Certificates for VMware vSphere Replication (2080395). Friedrich also has a great post on some of the basics – including the certificate replacement process – here.

VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 5

Posted on by

I’ve come across a few slightly odd things that I hadn’t accounted for during a recent vSphere 5.5 U2 deployment and thought it would be handy to document them. In this post (which is hopefully the last one) I’d like to cover off SSL certificates.

A lot of people don’t bother trying to deploy custom certificates because it invariably involves interaction with an in-house InfoSec team. This can be a royal pain in the arse. I understand completely. That said, getting custom certs into your vSphere environment has become a lot easier in recent times.

Firstly, there’s a few KB articles you should read:

Here’s the output from the Certificate Automation Tool

==================================================================
Main menu

Enter the action you want to run
   1. Plan your steps to update SSL certificates(Update Steps Planner)
   2. Generate Certificate Signing Requests
   3. Update Single Sign-On
   4. Update Inventory Service
   5. Update vCenter Server
   6. Update vCenter Orchestrator(vCO)
   7. Update vSphere Web Client and Log Browser
   8. Update vSphere Update Manager(VUM)
   9. End the update process and exit
The chosen action is: 1

And here’s what the Update Steps Planner gives you to work through.

The chosen action is: 1
==================================================================
1. Plan your steps to update SSL certificates(Update Steps Planner)

Choose the services you want to update:
      1. Single Sign-On
      2. Inventory Service
      3. vCenter Server
      4. vCenter Orchestrator
      5. vSphere Web Client
      6. Log Browser
      7. vSphere Update Manager
      8. All services(listed above)
      9. Return to the main menu

Example:
To choose the certificate update of Inventory Service, vCenter Server and vSphere Web Client you would enter: 2,3,5
You chose (enter comma-separated list of numbers): 8
Input arguments: [8]

Selected services: Single Sign-On, Inventory Service, vCenter Server, vCenter Orchestrator, Web Client, Log Browser, vSphere Update Manager
Detailed Plan to follow:
1. Go to the machine with Single Sign-On installed and - Update the Single Sign-On SSL certificate.
2. Go to the machine with Inventory Service installed and - Update Inventory Service trust to Single Sign-On.
3. Go to the machine with Inventory Service installed and - Update the Inventory Service SSL certificate.
4. Go to the machine with vCenter Server installed and - Update vCenter Server trust to Single Sign-On.
5. Go to the machine with vCenter Server installed and - Update the vCenter Server SSL certificate.
6. Go to the machine with vCenter Server installed and - Update vCenter Server trust to Inventory Service.
7. Go to the machine with Inventory Service installed and - Update the Inventory Service trust to vCenter Server.
8. Go to the machine with vCenter Orchestrator installed and - Update vCenter Orchestrator trust to Single Sign-On.
9. Go to the machine with vCenter Orchestrator installed and - Update vCenter Orchestrator trust to vCenter Server.
10. Go to the machine with vCenter Orchestrator installed and - Update the vCenter Orchestrator SSL certificate.
11. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to Single Sign-On.
12. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to Inventory Service.
13. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to vCenter Server.
14. Go to the machine with vSphere Web Client installed and - Update the vSphere Web Client SSL certificate.
15. Go to the machine with Log Browser installed and - Update the Log Browser trust to Single Sign-On.
16. Go to the machine with Log Browser installed and - Update the Log Browser SSL certificate.
17. Go to the machine with vSphere Update Manager installed and - Update the vSphere Update Manager SSL certificate.
18. Go to the machine with vSphere Update Manager installed and - Update vSphere Update Manager trust to vCenter Server.

And then you have a nice list of stuff to work through. I’m not going to dump the whole process here, but here’s a grab of what updating your vCenter cert looks like.

==================================================================
Main menu

Enter the action you want to run
   1. Plan your steps to update SSL certificates(Update Steps Planner)
   2. Generate Certificate Signing Requests
   3. Update Single Sign-On
   4. Update Inventory Service
   5. Update vCenter Server
   6. Update vCenter Orchestrator(vCO)
   7. Update vSphere Web Client and Log Browser
   8. Update vSphere Update Manager(VUM)
   9. End the update process and exit

The chosen action is: 5
==================================================================
5. Update the vCenter Server SSL Certificate

     1. Update the vCenter Server Trust to Single Sign-On
     2. Update the vCenter Server SSL Certificate
     3. Update the vCenter Server Trust to Inventory Service
     4. Rollback to the previous vCenter Server SSL Certificate
     5. Return to the main menu to update other services

The chosen service is: 2
[Thu 28/05/2015 - 10:39:54.86]: The services that are restarted as a part of this operation are: VMware VirtualCenter Server, VMware VirtualCenter Management Webservices and VMware vSphere Profile-Driven Storage Service.
Enter location to the new vCenter Server SSL chain: C:\Install\ssl-certificate-updater-tool-1308332\vCenterServer-VC4002\chain.pem
Enter location to the new vCenter Server private key: C:\Install\ssl-certificate-updater-tool-1308332\vCenterServer-VC4002\rui.key
Enter vCenter Server administrator user name: domain\svc_vmware
Enter vCenter Server administrator password (will not be echoed):
"Important: Enter the password carefully. The Certificate Automation Update Tool does not check the validity of the vCenter Server database password."
"A blank or incorrect password will leave the system in an inconsistent state, which will cause the vCenter Server to become unavailable. "
"If the system becomes unstable due to a bad password, see the Troubleshooting Section of KB 2041600."
Enter the vCenter Server original database password (will not be echoed):
Enter Single Sign-On Administrator user: Administrator@vsphere.local
Enter Single Sign-On Administrator password (will not be echoed):
[.] WARNING: Certificate's `CN=VC4002.racqgroup.local, OU=vCenterServer-VC4002, O=Company, L=Location, ST=QLD, C=AU' signature uses weak one-way h
ash (SHA-1). In a secure environment it is recommended to use SHA2-256 or a stronger hash algorithm.
[.] The supplied certificate chain is valid.
Loading 'screen' into random state - done
"Restarting services... (This can take some time)"
"Stopping vCenter Web Services..."
"Stopping vCenter Server..."
"Starting vCenter Server and other services..."
[Thu 28/05/2015 - 10:45:42.32]: Last operation update vCenter Server SSL certificate completed successfully.
[Thu 28/05/2015 - 10:45:42.33]: Go to the next step in the plan that was received from Update Steps Planner.

Once you’ve had your way with vCenter, etc, you can do your ESXi hosts. The following link has info on that – Configuring CA signed certificates for ESXi 5.x hosts, and you can grab the appropriate version of Win32 OpenSSL from here. Here’s what it looks like when you use OpenSSL to generate the requests for your ESXi hosts.

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.
C:\Users\Player1>cd \
C:\>cd OpenSSL\bin
C:\OpenSSL\bin>openssl req -new -nodes -out rui.csr -keyout rui-orig.key -config
openssl.cfg
Loading 'screen' into random state - done
Generating a 2048 bit RSA private key
........+++
..........................................+++
writing new private key to 'rui-orig.key'
-----
C:\OpenSSL\bin>openssl rsa -in rui-orig.key -out rui.key
writing RSA key
C:\OpenSSL\bin>

One thing to note. I found that HA got a bit irritable until all hosts in the cluster had custom certs installed. So it’s worth turning HA off until you’re finished. If, for some reason something goes wrong wit the ESXi certs, you can re-generate the default self-signed ones with the following command:

/sbin/generate-certificates

 

Updates In some of my previous posts, I talked about a few things that I had to do to get things working. In this post, I discussed the “Missing VMware Tools ISO”. I still don’t know why the tools files were missing from the installation, but I do know that once we applied some more recent vSphere Update Manager baselines to those hosts the correct ISO files were added to the hosts.

I also covered “HP Legacy BIOS Mode and ESXi” in this post. Interestingly, you’ll need to change back to UEFI BIOS mode if you’re trying to make VirtualConnect changes to a host, as my client found out the hard way.

I also spoke about ESXi hosts and Active Directory authentication in this post. I should point out that this post by Joseph also came in handy. If you find that when you restart the services on the host it bombs out, you’ll need to manually create /var/lock/subsys. There’s a KB article from VMware that says the same thing here.

mkdir /var/lock/subsys
/etc/init.d/netlogond restart
/etc/init.d/lwiod restart
/etc/init.d/lsassd restart

And you should then be right.

VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 4

Posted on by

I’ve been covering a few workarounds, mishaps and random things I’ve had to do during a recent vSphere 5.5 U2 deployment. This is Part 4 in the series, and I hope some of it is useful. You can read my other posts here, here, and here.

 

Client Integration Plug-in for vSphere Web Client 

Love it or hate it, the vSphere Web Client is here to stay. If, for some reason, you’re logged into a host with credentials that you want to use to log in to your vSphere environment with, you can use pass-through authentication if you install the Client Integration Plug-in for vSphere Web Client. You can get details on how to do that here.

 

HP Legacy BIOS Mode and ESXi

This was my first time using BL460c Generation 9 blades with ESXi. While I’ve been around the block with HP blades in the past, I’ve never used them with the SD card option before. I thought this was to blame when I’d reboot the hosts and configuration items (such as persistent scratch location, syslog configuration and core dump details) would disappear. Added to this, the networking configuration on vmk0 would disappear from time to time as well. I was also getting errors such as this when applying host profiles to hosts:

“Call “HostProfileManager.GenerateConfigTaskList” for object “HostProfileManager” on vCenter Server “hostname.domain” failed.

Failed to execute command to configure or query coredump partition.”

I did some searching and chanced upon this article BL460c Gen9 + ESXi 5.5 – Special procedure when using UEFI? Seems that setting the host’s Boot Mode to Legacy BIOS Mode makes for a happier installation and on-going experience. The guy who installed the blades had set them to Legacy mode for the installation and then set them back to UEFI. I can’t tell you why this needed to occur, nor can I tell you the disadvantages of taking this approach.

 

HP_BIOS_edit

 

Windows 2012 R2 and .Net 3.5 

If you’re running your VMware applications on Windows 2012 R2, there’s a chance you’ll need to install .Net 3.5 on your guest to get things working. This is handled via Server Roles. Microsoft has a TechNet article on how to do it here. Note that you’ll need your Windows installation media, and you’ll likely need to specify an alternate source – %CDROM%\sources\sxs.

 

specify_alternate_path_dot_net_35

 

Okay, so hopefully that was useful for someone. More to follow …

 

 

VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 3

Posted on by

This is my third post in a series of articles on some workarounds and things I had to look into when doing a recent vSphere 5.5 U2 deployment. You In can find my previous articles here and here. In this episode I’m covering EVC, Host Profile Compliance Checks and ESXi Hosts and Active Directory Authentication.

 

VMware Enhanced vMotion Compatibility (EVC)

Wondering which EVC Intel mode (Merom, Penryn, Nehalem, Westmere, Sandy Bridge, Ivy Bridge) to use with your vCenter cluster? It depends. This KB article provides a good outline of your options. Note that in vCenter Server 5.1 and 5.5, the Intel “Ivy Bridge” Generation option is only displayed in the Web Client. That’s the man trying to keep you down ;)

 

But how do I set EVC on the cluster when vCenter is virtual and running in the cluster? As the cluster is no longer the boundary for vMotion, one way to do this is to create a new empty cluster. Add your first host and setup as appropriate. Then enable EVC and vMotion the first guest into the cluster and you’ll be good to go. If you can’t vMotion across clusters because VMs are using various features of the CPU (a more likely scenario), you’ll need to use the method outlined in the following article – Enabling EVC on a cluster when vCenter Server is running in a virtual machine. It’s a bit of a pain, particularly if you’re using Distributed vSwitch, but it works well enough. And when VMware say they recommend you change your VM to standard vSwitch – it’s a good idea to take their advice.

 

VMware also have a pretty useful FAQ on EVC and CPU Compatibility that you can access here.

 

Host Profile Compliance Checks

If you’re running Host Profiles at the cluster level, you may find that even if the host is compliant, it fails on Fault Tolerance checks. If you’re not using FT, disable those checks. Because we all live for green lights. Right-click the cluster and click Edit Settings > VMware HA > Advanced Options. You’ll need to add in a field and set it to false. Details on how to do this can be found here.

 

ESXi Hosts and Active Directory Authentication

Want to join your ESXi host to an Active Directory domain? Good idea. You’ll need this KB article. Be sure you’ve got Config.HostAgent.plugins.hostsvc.esxAdminsGroup set correctly, or you’ll have a difficult time getting in with your AD credentials. If you have issues, you can try restarting LDAP or forcing an update on the DC that you configured the ESXi host to look at. I found this article useful.

 

Okay, so hopefully that was useful for someone. More to follow …

VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 2

Posted on by

In a previous post, I mentioned I was doing a vSphere deployment sitting on some HP blades with SD cards installed. Because of this, I had to configure a few different parts of ESXi to point to network services rather than using local, persistent storage. This isn’t a bad thing in any case, as you really want your logs and core dumps to be off-host when you’re trying to troubleshoot host issues.

 

Network Dump Collector

It’s a good idea to have your core dumps going to a central location. you can do this with Network Dump Collector or via shared diagnostic storage. In this instance, I’ve provided an example on how to configure the host to use the Network Dump Collector, which can be installed from the vCenter installation media.

 

login as: root
 Using keyboard-interactive authentication.
 Password:
 The time and date of this login have been sent to the system logs.

VMware offers supported, powerful system administration tools.  Please
 see www.vmware.com/go/sysadmintools for details.

The ESXi Shell can be disabled by an administrative user. See the
 vSphere Security documentation for more information.
 ~ # esxcli system coredump network get
 Enabled: false
 Host VNic:
 Network Server IP:
 Network Server Port: 0
 ~ # esxcli system coredump network set --interface-name vmk0 --server-ipv4 192.168.0.100 --server-port 6500
 ~ # esxcli system coredump network set --enable true
 ~ # esxcli system coredump network get
 Enabled: true
 Host VNic: vmk0
 Network Server IP: 192.168.0.100
 Network Server Port: 6500
 ~ # esxcli system coredump network check
 Verified the configured netdump server is running
 ~ # /sbin/auto-backup.sh
 Files /etc/vmware/dvsdata.db and /tmp/auto-backup.1942245//etc/vmware/dvsdata.db differ
 Saving current state in /bootbank
 Clock updated.
 Time: 05:04:38   Date: 04/21/2015   UTC
 ~ #

 

It’s important to check that the Network Dump Collector service is running, too.

 

vSphere Syslog Server

You’ll also want to point your syslogs to a remote location. If you don’t have access to syslog in your environment, you can set it up from the vCenter installation media. Here’s an article on how to do that. Here’re the rough steps you need to take.

Check your current configuration first.

~ # esxcli system syslog config get
 Default Network Retry Timeout: 180
 Local Log Output: /scratch/log
 Local Log Output Is Configured: false
 Local Log Output Is Persistent: false
 Local Logging Default Rotation Size: 1024
 Local Logging Default Rotations: 8
 Log To Unique Subdirectory: false
 Remote Host: <none>

Now you can set the configuration.

~ # esxcli system syslog config set --loghost='tcp://192.168.0.100:514'

Load the configuration.

~ # esxcli system syslog reload

Check that the configuration worked.

~ # esxcli system syslog config get

Backup the configuration.

~ # /sbin/auto-backup.sh

 

Persistent Scratch

I recommend setting a persistent scratch location as well. You can read about that here. In my case I used shared storage with directories for each host. I then set that as an option to configure via Host Profiles.

 

Okay, so hopefully that was useful for someone. More to follow …

 

VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 1

Posted on by

I’ve been in the field recently, deploying one of my designs for a customer as our normal resources were tied up elsewhere. I’m rusty, to be sure, and haven’t done a lot of VMware deployment work outside of a lab in the last few years. So I thought I’d do a series of posts about things that I didn’t think of when I did the design, and the various workarounds and random fixes that I came across to resolve the issues I encountered. I hope some of them are useful. I’ll break this up into a couple of parts, covering a few different things in each post.

 

Missing VMware Tools ISO

The first issue I came across was being unable to load the VMware Tools ISO when updating or installing VMware Tools on a VM. You’ll need the following VMware KB article – Unable to locate the ISO image for the VMware Tools installation (1036810) – to resolve the issue. I literally have no effing idea why this is happening. I can only assume that it has something to do with the fact we’ve used SD cards in these blades, and ESXi thinks, quite rightly, that the local storage on these blades is not so persistent.

 

Moving vCenter’s SQL Database

The Client also had made some SQL servers available as per my request prior to arriving on-site. However, The Client wanted me to build vCenter on a “temporary” SQL host and then migrate it elsewhere. I’ve found that life is not always as we planned it. If this happens to you, you’ll find the following article – Moving the VMware vCenter Server 4.x/5.x/6.0.x SQL database (7960893) – to be an invaluable. You should also get friendly with your local SQL DBA if you’re lucky to have one at hand. Again, I could talk about how planning and preparation are key to a successful engagement, but the reality is that sometimes The Client has outside pressures applied to them that I simply can’t appreciate.

 

ESXi NIC Teaming Options

Found yourself in a data centre? Touching a vSphere deployment? Teaming NICs in the dark of night? Not really knowing what’s going on? Wondering what all those options meant? Wonder no more. NIC teaming in ESXi and ESX (1004088) covers a lot of useful information, including the following summary of your options when it comes to uplinks:

  • Route based on the originating port ID: Choose an uplink based on the virtual port where the traffic entered the virtual switch.
  • Route based on an IP hash: Choose an uplink based on a hash of the source and destination IP addresses of each packet. For non-IP packets, whatever is at those offsets is used to compute the hash.
  • Route based on a source MAC hash: Choose an uplink based on a hash of the source Ethernet.
  • Use explicit failover order: Always use the highest order uplink from the list of Active adapters which passes failover detection criteria.
  • Route based on physical NIC load (Only available on Distributed Switch): Choose an uplink based on the current loads of physical NICs.

The impact of choosing the different options here is worthy of a future blog post, but in the meantime, have a look at this handy article at vLore Blog.

 

Okay, so hopefully that was useful for someone. More to follow …

VMware – vSphere 5.x License Editions and Features

Posted on by

In the spirit of letting those around me do all the hard work to generate content for this blog, my colleague was asking about limitations on various editions of vSphere 5.1. I was vague with the answer, but to his credit, he kept looking. Turns out, someone’s asked this question before, and VMware have even written a KB about it (2001113). It’s worth looking at, particularly when you’re moving from 5.1 – 5.5.

 

VMware – Creating a 32bit DSN on a 64bit Windows machine (1010401)

Posted on by

I know I’ve covered this somewhere before – but this is just a reminder post for my own reference. When I was building my lab on vSphere 5.1 I had to configure a 32-bit DSN for VMware Update Manager- this link provides details on how to do it. I believe this is no longer necessary with 5.5. Again, not very exciting, but useful to know how to do it. Also, statements like “You can install the Update Manager server only on 64bit machines. However, Update Manager is a 32bit application and requires a 32bit DSN” still blow my mind.