VMware – vSphere Replication 5.8 and Custom Certificates

Posted on 2015-06-16 by dan

I waffled on some time ago about using proper certificates in your vSphere 5.5 environment. You can read about some of how to do that here. Eric has a nice summary of the steps here. I got a call recently from the customer about a few things and they mentioned some issues with vSphere Replication 5.8. Turns out I’d forgotten about vSphere Replication when I’d gone through the certificate replacement process, as it was done as a PoC. The fix is simple: power off the appliance and power it on again. VMware has a KB for most every situation, including this one – VMware vSphere Replication appliance no longer able to communicate with the VMware vCenter Server after changing the vCenter certificates (2063955). It also helps that I’m a bit late to this particular party.

The next step should be to replace the certificates on your vSphere Replication infrastructure as well. I was going to put together a post on that too, but it’s probably simplest if you read the VMware KB – Configuring CA Signed Certificates for VMware vSphere Replication (2080395). Friedrich also has a great post on some of the basics – including the certificate replacement process – here.

VMware – vSphere 5.5 U2 – Resetting the SSO Password

Posted on 2015-06-01 by dan

In a previous post I talked about deploying custom SSL certs into a vCenter 5.5 environment. As I was working through the update steps, the Certificate Automation Tool kept bombing out when updating the Inventory Service certificate. Neither the client nor I really knew why this was happening, but I had a bit of a hunch that it something to do with SSO credentials. It turned out to be a lucky guess, as I reset the password a few times and the SSL cert tool started working.

If you find yourself in this situation, there’s a tool provided with vCenter to reset the SSO password. Here’s a link to the KB article.

c:\Program Files\VMware\Infrastructure\VMware\CIS\vmdird>vdcadmintool.exe

It’s a fairly straightforward process, but you need to be mindful to use a generated password that meets VMware’s requirements for SSO passwords and special characters. By that I mean that some special characters aren’t allowed, even though they’re in passwords generated by the tool. You can get details on that here. In short, these special characters are not supported in SSO passwords:

Non-ASCII characters
Ampersand (&)
Semicolon ( ; )
Double quotation mark ( ” )
Single quotation mark ( ‘ )
Circumflex ( ^ )
Backslash ( \ )
Percentage (%)

At times I wasn’t convinced that this list is comprehensive either.

VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 5

Posted on 2015-05-31 by dan

I’ve come across a few slightly odd things that I hadn’t accounted for during a recent vSphere 5.5 U2 deployment and thought it would be handy to document them. In this post (which is hopefully the last one) I’d like to cover off SSL certificates.

A lot of people don’t bother trying to deploy custom certificates because it invariably involves interaction with an in-house InfoSec team. This can be a royal pain in the arse. I understand completely. That said, getting custom certs into your vSphere environment has become a lot easier in recent times.

Firstly, there’s a few KB articles you should read:

Here’s the output from the Certificate Automation Tool

==================================================================
Main menu

Enter the action you want to run
   1. Plan your steps to update SSL certificates(Update Steps Planner)
   2. Generate Certificate Signing Requests
   3. Update Single Sign-On
   4. Update Inventory Service
   5. Update vCenter Server
   6. Update vCenter Orchestrator(vCO)
   7. Update vSphere Web Client and Log Browser
   8. Update vSphere Update Manager(VUM)
   9. End the update process and exit
The chosen action is: 1

And here’s what the Update Steps Planner gives you to work through.

The chosen action is: 1
==================================================================
1. Plan your steps to update SSL certificates(Update Steps Planner)

Choose the services you want to update:
      1. Single Sign-On
      2. Inventory Service
      3. vCenter Server
      4. vCenter Orchestrator
      5. vSphere Web Client
      6. Log Browser
      7. vSphere Update Manager
      8. All services(listed above)
      9. Return to the main menu

Example:
To choose the certificate update of Inventory Service, vCenter Server and vSphere Web Client you would enter: 2,3,5
You chose (enter comma-separated list of numbers): 8
Input arguments: [8]

Selected services: Single Sign-On, Inventory Service, vCenter Server, vCenter Orchestrator, Web Client, Log Browser, vSphere Update Manager
Detailed Plan to follow:
1. Go to the machine with Single Sign-On installed and - Update the Single Sign-On SSL certificate.
2. Go to the machine with Inventory Service installed and - Update Inventory Service trust to Single Sign-On.
3. Go to the machine with Inventory Service installed and - Update the Inventory Service SSL certificate.
4. Go to the machine with vCenter Server installed and - Update vCenter Server trust to Single Sign-On.
5. Go to the machine with vCenter Server installed and - Update the vCenter Server SSL certificate.
6. Go to the machine with vCenter Server installed and - Update vCenter Server trust to Inventory Service.
7. Go to the machine with Inventory Service installed and - Update the Inventory Service trust to vCenter Server.
8. Go to the machine with vCenter Orchestrator installed and - Update vCenter Orchestrator trust to Single Sign-On.
9. Go to the machine with vCenter Orchestrator installed and - Update vCenter Orchestrator trust to vCenter Server.
10. Go to the machine with vCenter Orchestrator installed and - Update the vCenter Orchestrator SSL certificate.
11. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to Single Sign-On.
12. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to Inventory Service.
13. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to vCenter Server.
14. Go to the machine with vSphere Web Client installed and - Update the vSphere Web Client SSL certificate.
15. Go to the machine with Log Browser installed and - Update the Log Browser trust to Single Sign-On.
16. Go to the machine with Log Browser installed and - Update the Log Browser SSL certificate.
17. Go to the machine with vSphere Update Manager installed and - Update the vSphere Update Manager SSL certificate.
18. Go to the machine with vSphere Update Manager installed and - Update vSphere Update Manager trust to vCenter Server.

And then you have a nice list of stuff to work through. I’m not going to dump the whole process here, but here’s a grab of what updating your vCenter cert looks like.

==================================================================
Main menu

Enter the action you want to run
   1. Plan your steps to update SSL certificates(Update Steps Planner)
   2. Generate Certificate Signing Requests
   3. Update Single Sign-On
   4. Update Inventory Service
   5. Update vCenter Server
   6. Update vCenter Orchestrator(vCO)
   7. Update vSphere Web Client and Log Browser
   8. Update vSphere Update Manager(VUM)
   9. End the update process and exit

The chosen action is: 5
==================================================================
5. Update the vCenter Server SSL Certificate

     1. Update the vCenter Server Trust to Single Sign-On
     2. Update the vCenter Server SSL Certificate
     3. Update the vCenter Server Trust to Inventory Service
     4. Rollback to the previous vCenter Server SSL Certificate
     5. Return to the main menu to update other services

The chosen service is: 2
[Thu 28/05/2015 - 10:39:54.86]: The services that are restarted as a part of this operation are: VMware VirtualCenter Server, VMware VirtualCenter Management Webservices and VMware vSphere Profile-Driven Storage Service.
Enter location to the new vCenter Server SSL chain: C:\Install\ssl-certificate-updater-tool-1308332\vCenterServer-VC4002\chain.pem
Enter location to the new vCenter Server private key: C:\Install\ssl-certificate-updater-tool-1308332\vCenterServer-VC4002\rui.key
Enter vCenter Server administrator user name: domain\svc_vmware
Enter vCenter Server administrator password (will not be echoed):
"Important: Enter the password carefully. The Certificate Automation Update Tool does not check the validity of the vCenter Server database password."
"A blank or incorrect password will leave the system in an inconsistent state, which will cause the vCenter Server to become unavailable. "
"If the system becomes unstable due to a bad password, see the Troubleshooting Section of KB 2041600."
Enter the vCenter Server original database password (will not be echoed):
Enter Single Sign-On Administrator user: Administrator@vsphere.local
Enter Single Sign-On Administrator password (will not be echoed):
[.] WARNING: Certificate's `CN=VC4002.racqgroup.local, OU=vCenterServer-VC4002, O=Company, L=Location, ST=QLD, C=AU' signature uses weak one-way h
ash (SHA-1). In a secure environment it is recommended to use SHA2-256 or a stronger hash algorithm.
[.] The supplied certificate chain is valid.
Loading 'screen' into random state - done
"Restarting services... (This can take some time)"
"Stopping vCenter Web Services..."
"Stopping vCenter Server..."
"Starting vCenter Server and other services..."
[Thu 28/05/2015 - 10:45:42.32]: Last operation update vCenter Server SSL certificate completed successfully.
[Thu 28/05/2015 - 10:45:42.33]: Go to the next step in the plan that was received from Update Steps Planner.

Once you’ve had your way with vCenter, etc, you can do your ESXi hosts. The following link has info on that – Configuring CA signed certificates for ESXi 5.x hosts, and you can grab the appropriate version of Win32 OpenSSL from here. Here’s what it looks like when you use OpenSSL to generate the requests for your ESXi hosts.

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.
C:\Users\Player1>cd \
C:\>cd OpenSSL\bin
C:\OpenSSL\bin>openssl req -new -nodes -out rui.csr -keyout rui-orig.key -config
openssl.cfg
Loading 'screen' into random state - done
Generating a 2048 bit RSA private key
........+++
..........................................+++
writing new private key to 'rui-orig.key'
-----
C:\OpenSSL\bin>openssl rsa -in rui-orig.key -out rui.key
writing RSA key
C:\OpenSSL\bin>

One thing to note. I found that HA got a bit irritable until all hosts in the cluster had custom certs installed. So it’s worth turning HA off until you’re finished. If, for some reason something goes wrong wit the ESXi certs, you can re-generate the default self-signed ones with the following command:

/sbin/generate-certificates

Updates In some of my previous posts, I talked about a few things that I had to do to get things working. In this post, I discussed the “Missing VMware Tools ISO”. I still don’t know why the tools files were missing from the installation, but I do know that once we applied some more recent vSphere Update Manager baselines to those hosts the correct ISO files were added to the hosts.

I also covered “HP Legacy BIOS Mode and ESXi” in this post. Interestingly, you’ll need to change back to UEFI BIOS mode if you’re trying to make VirtualConnect changes to a host, as my client found out the hard way.

I also spoke about ESXi hosts and Active Directory authentication in this post. I should point out that this post by Joseph also came in handy. If you find that when you restart the services on the host it bombs out, you’ll need to manually create /var/lock/subsys. There’s a KB article from VMware that says the same thing here.

mkdir /var/lock/subsys
/etc/init.d/netlogond restart
/etc/init.d/lwiod restart
/etc/init.d/lsassd restart

And you should then be right.

VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 4

Posted on 2015-04-28 by dan

I’ve been covering a few workarounds, mishaps and random things I’ve had to do during a recent vSphere 5.5 U2 deployment. This is Part 4 in the series, and I hope some of it is useful. You can read my other posts here, here, and here.

Client Integration Plug-in for vSphere Web Client

Love it or hate it, the vSphere Web Client is here to stay. If, for some reason, you’re logged into a host with credentials that you want to use to log in to your vSphere environment with, you can use pass-through authentication if you install the Client Integration Plug-in for vSphere Web Client. You can get details on how to do that here.

HP Legacy BIOS Mode and ESXi

This was my first time using BL460c Generation 9 blades with ESXi. While I’ve been around the block with HP blades in the past, I’ve never used them with the SD card option before. I thought this was to blame when I’d reboot the hosts and configuration items (such as persistent scratch location, syslog configuration and core dump details) would disappear. Added to this, the networking configuration on vmk0 would disappear from time to time as well. I was also getting errors such as this when applying host profiles to hosts:

“Call “HostProfileManager.GenerateConfigTaskList” for object “HostProfileManager” on vCenter Server “hostname.domain” failed.

Failed to execute command to configure or query coredump partition.”

I did some searching and chanced upon this article BL460c Gen9 + ESXi 5.5 – Special procedure when using UEFI? Seems that setting the host’s Boot Mode to Legacy BIOS Mode makes for a happier installation and on-going experience. The guy who installed the blades had set them to Legacy mode for the installation and then set them back to UEFI. I can’t tell you why this needed to occur, nor can I tell you the disadvantages of taking this approach.

Windows 2012 R2 and .Net 3.5

If you’re running your VMware applications on Windows 2012 R2, there’s a chance you’ll need to install .Net 3.5 on your guest to get things working. This is handled via Server Roles. Microsoft has a TechNet article on how to do it here. Note that you’ll need your Windows installation media, and you’ll likely need to specify an alternate source – %CDROM%\sources\sxs.

Okay, so hopefully that was useful for someone. More to follow …

VMware – vExpert 2015

Posted on 2015-02-06 by dan

I’m very excited and humbled to have been listed as a vExpert for 2015. This is the third year in a row that they’ve forgotten to delete my name off the list (BTW I’ll never get tired of that joke). The announcement and list is here, and I expect more news about this year’s programme soon. Thanks again to Corey Romero and the rest of the VMware Social people for making this kind of thing happen. And thanks also to the vExpert community for being, well, such a neat (and growing) community to be part of.

VMware – vExpert 2014

Posted on 2014-04-02 by dan

I’m very excited and humbled to have been listed as a vExpert 2014. This is the second year in a row that they’ve forgotten to delete my name. The announcement and list is here, and I expect more news about it soon. Thanks again to John Mark Troyer, Corey Romero and the rest of the VMware Social people for making this kind of thing happen. And thanks also to the vExpert community for being, well, such a neat community to be part of. Q2 applications are open at http://bit.ly/LMJqB5.

*Update*

The nice folks at Login VSI are offering a free 12 month license to vExperts, Microsoft MVPs and Citrix Technology Professionals. You can get more info about it here.

I was also stoked to hear that Pluralsight are offering a year of free training to vExperts and Microsoft MVPs as well. Check out the post here.

*Update2*

The friendly people at Tintri are also once again stumping up for shirts too. You can get more info here.

VMware vExpert 2013

Posted on 2013-06-05 by dan

I’m a bit behind at the moment, but I wanted to take a moment to say thank you to John Troyer, Corey Romero and the folks in the VMware Social Media & Community Team for leaving me on the list for vExpert 2013. It’s a long list, and you’ll notice a lot of familiar names on there. So, thanks.

It’s also been really cool to see the rest of the community kick in with some goodies for vExperts. TrainSignal have come through with a year’s free training for vExperts – which I think is really quite awesome. You can also sign up for a free trial here – worth checking out.