VMware – vSphere Basics – Re-package An OVA

This is a quick and easy one. I came across a virtual appliance the other day that had an expired certificate.

When you click Next you’ll get an error saying the package is signed with an invalid certificate.

It’s a relatively easy fix (or at least workaround) and I followed Stephen Wagner‘s guidance here. In short, grab a copy of the VMware OVF Tool from here. You then run the following command:

ovftool.exe --skipManifestCheck c:\tmp\old.ova c:\tmp\new.ova

You’ll then be able to deploy the appliance without it barfing. Remember, though, that this is a bit of a rough workaround, and you should really contact the appliance vendor in the first instance as they’ll likely be keen to fix the issue. In my case I was able to continue with my testing while the vendor went ahead and fixed things on their side.

VMware – vSphere Basics – Create a Custom Role

I’ve been evaluating a data protection solution in the lab recently and wanted to create a custom role in vCenter for the solution to use. It’s a basic thing, but if you don’t do it often it might not be that obvious where to click. The VMware documentation site has more information on creating a custom role as well. Why would you do this? In the same way it’s a bad idea to give every service Domain Administrator privileges, it’s also a bad idea to give your data protection solutions elevated privileges in your environment. If you’re into that kind of thing, read this guidance on roles and permissions too. In this example, I created a “CohesityTest” user as a domain user in Active Directory. I then wanted to assign that user to a custom role in vCenter and assign it certain privileges. In this example I’m using vCenter 6.5 with the Web Client. The process is as follows.

Go to the Home screen in vCenter and click on “Administration”.

In this example, I’ve already created a Role called Cohesity (following the instructions above) and assigned privileges to that Role.

Click on “Global Permissions” and the click on the green plus sign.

I want to add a user to a role. Click on “Add”.

The user I want to add is a domain user, so I use the drop down box to select the domain the user resides in.

Typing “coh” into the search field yields the only user with those letters in their name.

Once the user is selected, you can click on “Add” and then “OK”.

Make sure the user has the appropriate Role assigned. In this example, I’m assigning the CohesityTest user to the Cohesity Role and propagating these changes to child objects. Click “OK”. And then you’re done.

To check your role has the correct privileges, click on “Roles”, “Role Name”, and then “Privileges” and you can expand the items to check the correct privileges are assigned.

Once I’d done this I went back and re-added the vCenter to the Cohesity environment using the CohesityTest user and I was good to go.

VMware – vSphere Basics – vCenter 6.5 Upgrade Scenarios

I did an article on the vSphere 6 Platform Services Controller a while ago. After attending a session on changes in vSphere 6.5 at vFORUM, I thought it would be an idea to revisit this, and frame it in the context of vCenter 6.5 upgrades.

 

vSphere Components

In vCenter 6.5, the architecture is a bit different to 5.x. With the PSC, you get:

  • VMware vCenter Single Sign-On
  • License service
  • Lookup service
  • VMware Directory Services
  • VMware Certificate Authority

And the vCenter Server Service gives you:

  • vCenter Server
  • VMware vSphere Web Client
  • VMware vSphere Auto Deploy
  • VMware vSphere ESXi Dump Collector
  • vSphere Syslog Collector on Windows and vSphere Syslog Service for VMware vCenter Server Appliance
  • vSphere Update Manager

 

Architecture Choices

There are some basic configurations that you can go with, but I generally don’t recommend these for anything outside of a lab or test environment. In these configurations, the PSC is either embedded or external to the vCenter Server. The choice here will be dependent on the sizing and feature requirements of your environment.

If you want to use Enhanced Linked Mode an external PSC is recommended. If you want it highly available, you’ll still need to use a load balancer. This VMware KB  article provides some handy insights and updates from 6.0.

 

vCenter Upgrade Scenarios

Your upgrade architecture you’ll choose depends on where your vCenter services reside. If your vCenter server has SSO installed, it becomes a vCenter Server with an embedded PSC.

If, however, some of the vSphere components are installed on separate VMs then the Web Client and Inventory Service become part of the “Management Node” (your vCenter box) and the PSC (with SSO) is separate/external.

Note also that vSphere 6.5 still requires a load balancer for vSphere High Availability.

 

Final Thoughts

This is not something that’s necessarily going to come up each day. But if you’re working either directly with VMware, via an integrator or doing it yourself, your choice of vCenter architecture should be a key consideration in your planning activities. As with most upgrades to key infrastructure components, you should take the time to plan appropriately.

VMware – vSphere 6 Basics – Platform Services Controller

I’ve finally gotten some time to dig into the changes in vSphere 6 with regards to deployment options and architecture. I thought I’d do a few posts covering some key enhancements from VMware, paying particular attention to the Platform Service Controller (PSC) and VMware’s preferred deployment options. I haven’t received any briefings from VMware, so I can’t comment on what is coming in future releases. Note that most of this information was made available to me via access to VMware’s partner program, and I think it’s important that more people understand what’s going on when it comes to PSC and how it works.

 

vSphere Components

The PSC is a new feature in vSphere 6.0. As background, I recommend you first check out this blog post – vCenter Server 6 Deployment Topologies and High Availability. There is also an excellent FAQ from VMware available here. I thought, before diving too much into PSC deployment options, it’s a good idea to revisit VMware’s semi-new approach to vSphere components.

The PSC contains the following services:

  • VMware vCenter Single Sign-On (SSO);
  • License Service;
  • Lookup Service;
  • VMware Directory Service; and
  • VMware Certificate Authority (CA).

Everything else is now referred to as “vCenter Services”, providing the remainder of the vCenter Server functionality.  This includes:

  • vCenter Server;
  • VMware vSphere Web Client;
  • Inventory Service;
  • vSphere Auto Deploy;
  • VMware vSphere ESXi Dump Collector; and
  • VMware vSphere Syslog Collector (Windows) / VMware Syslog Service (Appliance).

 

Enhanced Linked Mode and PSC Deployment Options

Here are a few different ways you can do it. Some are good, some are bad. VMware has published a list of recommended topologies for VMware vSphere 6.0.x. The following section provides an overview of the options. Note that some of these options aren’t without their issues.

 

Enhanced Linked Mode with an External PSC Without HA

The PSC is configured on a separate VM and then the vCenter Servers are joined to that domain, providing Enhanced Linked mode functionality.

ELM1

 

Enhanced Linked Mode with an External PSC in an HA Configuration

In this case, the PSCs are configured on separate VMs behind a load balancer to provide HA for the configuration. The vCenter Servers are then joined to that domain using the shared load balancer IP address, providing Enhanced Linked mode functionality that is fault-tolerant.

ELM2

And here’s a few ways that you can do it that aren’t really recommended.

 

Enhanced Linked Mode with Embedded PSCs (Not Recommended)

In this scenario, vCenter is installed in an embedded configuration on the first server. Subsequent installations are then configured in embedded mode but joined to an existing SSO domain. Linking the embedded PSCs is possible, but VMware does not recommend this configuration.

ELM3

 

Enhanced Linked Mode in Combination Deployment (Not Recommended)

In a combination deployment, the embedded and external PSC architectures are combined. While linking an embedded PSC and an external PSC is possible, VMware does not recommended this configuration.

ELM4

 

Enhanced Linked Mode using only an Embedded PSC (Not Recommended)

In this case there is an embedded PSC and vCenter Server linked with an external standalone vCenter Server. Linking a second vCenter Server to an existing embedded vCenter Server and PSC is possible, but VMware does not recommended this configuration.

ELM5

 

Sizing Considerations

If you’re not going to use enhanced linked mode, use an embedded PSC. You still have availability via VMware HA. The failure domain is limited to a single vCenter Server, as there is no dependency on external component connectivity for PSC connectivity. This is most suitable for lab environments.

For sites that will use enhanced linked mode use external PSCs.  The number of controllers depends on the size of the environment:

  • Between 2 and 4 VMware solutions – a single PSC for no HA, and 2 will be required for HA configured behind a single load balancer.
  • Between 4 and 8 VMware solutions – two PSCs linked together for no HA, and four will be required for HA configured behind two load balancers (two behind each load balancer).
  • Between 8 and 10 VMware solutions – three PSCs linked together for no HA, and six will be required for HA configured behind three load balancers (two behind each load balancer).

HA is provided by having multiple PSCs and a load balancer to provide failure protection. All components are still protected by VMware HA. This VMware KB has more information on how to set this up – Configuring PSC 6.0 High Availability for vSphere 6.0 using vCenter Server 6.0 Appliance.

 

vCenter Platform Choice

VMware maintain that, with the improvements to the vCenter appliance platform, the choice of Windows-based vs vCenter appliance is now a matter of preference rather than performance. I recommend the appliance wherever possible, but some people will feel more comfortable with a Windows-based platform. The cool thing is that, if you want to make things complicated, the PSC supports mixed-mode (i.e. appliance and Windows-based vCenter deployments).

PSC_mixed

 

Final Thoughts

This may have gone a bit beyond basics, and it’s not something that’s necessarily going to come up each day. But if you’re working either directly with VMware, via an integrator or doing it yourself, this new approach should be a key consideration in your planning activities. The addition of the PSC concept to the vCenter architecture improves the flexibility and availability options of the product, something that I think VMware has struggled with in the past. The key takeaway, in my opinion, is that if you’re upgrading from 5.5 or below, you need to take the time to plan appropriately, particularly if you want to leverage some of the new features that are available.

VMware – vSphere Replication 5.8 and Custom Certificates

I waffled on some time ago about using proper certificates in your vSphere 5.5 environment. You can read about some of how to do that here. Eric has a nice summary of the steps here. I got a call recently from the customer about a few things and they mentioned some issues with vSphere Replication 5.8. Turns out I’d forgotten about vSphere Replication when I’d gone through the certificate replacement process, as it was done as a PoC. The fix is simple: power off the appliance and power it on again. VMware has a KB for most every situation, including this one – VMware vSphere Replication appliance no longer able to communicate with the VMware vCenter Server after changing the vCenter certificates (2063955). It also helps that I’m a bit late to this particular party.

The next step should be to replace the certificates on your vSphere Replication infrastructure as well. I was going to put together a post on that too, but it’s probably simplest if you read the VMware KB – Configuring CA Signed Certificates for VMware vSphere Replication (2080395). Friedrich also has a great post on some of the basics – including the certificate replacement process – here.

VMware – SRM 5.8 – You had one job!

The Problem

A colleague of mine has been doing some data centre failover testing for a customer recently and ran into an issue with VMware’s Site Recovery Manager (SRM) 5.8 running on vSphere 5.5 U2. When attempting to perform a recovery, and you’re running Linked Mode, and the protected site is off-line, the recovery may fail. The upshot of this is “The user is unable to perform a recovery at the recovery site, in the event of a DR scenario”. Here’s what it looks like.

SRM1

 

The Reason and Resolution

You can read more about the problem in this VMware KB article: Performing a Recovery using the Web Client in VMware vCenter Site Recovery Manager 5.8 reports the error: Failed to connect Site Recovery Manager Server(s). In short, there’s a PowerShell script you can run to make the recovery happen.

SRM0

 

Conclusion

I don’t know what to say about this. I’d like to put the boot into whomever at VMware is responsible for this SNAFU, but I’m guessing that they’ve already had a hard time of it. At least, I guess, there’s a workaround, if not a fix. But you’d be a bit upset if this happened for the first time during a real failover. But that’s why we test before we handover. And what is it with everything going pear-shaped when Linked Mode is in use?

 

*Update – 29/10/2015*

Marcel van den Berg recently pointed out that updating to SRM 5.8.1 resolves this issue. Further detail can be found here.

VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 5

I’ve come across a few slightly odd things that I hadn’t accounted for during a recent vSphere 5.5 U2 deployment and thought it would be handy to document them. In this post (which is hopefully the last one) I’d like to cover off SSL certificates.

A lot of people don’t bother trying to deploy custom certificates because it invariably involves interaction with an in-house InfoSec team. This can be a royal pain in the arse. I understand completely. That said, getting custom certs into your vSphere environment has become a lot easier in recent times.

Firstly, there’s a few KB articles you should read:

Here’s the output from the Certificate Automation Tool

==================================================================
Main menu

Enter the action you want to run
   1. Plan your steps to update SSL certificates(Update Steps Planner)
   2. Generate Certificate Signing Requests
   3. Update Single Sign-On
   4. Update Inventory Service
   5. Update vCenter Server
   6. Update vCenter Orchestrator(vCO)
   7. Update vSphere Web Client and Log Browser
   8. Update vSphere Update Manager(VUM)
   9. End the update process and exit
The chosen action is: 1

And here’s what the Update Steps Planner gives you to work through.

The chosen action is: 1
==================================================================
1. Plan your steps to update SSL certificates(Update Steps Planner)

Choose the services you want to update:
      1. Single Sign-On
      2. Inventory Service
      3. vCenter Server
      4. vCenter Orchestrator
      5. vSphere Web Client
      6. Log Browser
      7. vSphere Update Manager
      8. All services(listed above)
      9. Return to the main menu

Example:
To choose the certificate update of Inventory Service, vCenter Server and vSphere Web Client you would enter: 2,3,5
You chose (enter comma-separated list of numbers): 8
Input arguments: [8]

Selected services: Single Sign-On, Inventory Service, vCenter Server, vCenter Orchestrator, Web Client, Log Browser, vSphere Update Manager
Detailed Plan to follow:
1. Go to the machine with Single Sign-On installed and - Update the Single Sign-On SSL certificate.
2. Go to the machine with Inventory Service installed and - Update Inventory Service trust to Single Sign-On.
3. Go to the machine with Inventory Service installed and - Update the Inventory Service SSL certificate.
4. Go to the machine with vCenter Server installed and - Update vCenter Server trust to Single Sign-On.
5. Go to the machine with vCenter Server installed and - Update the vCenter Server SSL certificate.
6. Go to the machine with vCenter Server installed and - Update vCenter Server trust to Inventory Service.
7. Go to the machine with Inventory Service installed and - Update the Inventory Service trust to vCenter Server.
8. Go to the machine with vCenter Orchestrator installed and - Update vCenter Orchestrator trust to Single Sign-On.
9. Go to the machine with vCenter Orchestrator installed and - Update vCenter Orchestrator trust to vCenter Server.
10. Go to the machine with vCenter Orchestrator installed and - Update the vCenter Orchestrator SSL certificate.
11. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to Single Sign-On.
12. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to Inventory Service.
13. Go to the machine with vSphere Web Client installed and - Update vSphere Web Client trust to vCenter Server.
14. Go to the machine with vSphere Web Client installed and - Update the vSphere Web Client SSL certificate.
15. Go to the machine with Log Browser installed and - Update the Log Browser trust to Single Sign-On.
16. Go to the machine with Log Browser installed and - Update the Log Browser SSL certificate.
17. Go to the machine with vSphere Update Manager installed and - Update the vSphere Update Manager SSL certificate.
18. Go to the machine with vSphere Update Manager installed and - Update vSphere Update Manager trust to vCenter Server.

And then you have a nice list of stuff to work through. I’m not going to dump the whole process here, but here’s a grab of what updating your vCenter cert looks like.

==================================================================
Main menu

Enter the action you want to run
   1. Plan your steps to update SSL certificates(Update Steps Planner)
   2. Generate Certificate Signing Requests
   3. Update Single Sign-On
   4. Update Inventory Service
   5. Update vCenter Server
   6. Update vCenter Orchestrator(vCO)
   7. Update vSphere Web Client and Log Browser
   8. Update vSphere Update Manager(VUM)
   9. End the update process and exit

The chosen action is: 5
==================================================================
5. Update the vCenter Server SSL Certificate

     1. Update the vCenter Server Trust to Single Sign-On
     2. Update the vCenter Server SSL Certificate
     3. Update the vCenter Server Trust to Inventory Service
     4. Rollback to the previous vCenter Server SSL Certificate
     5. Return to the main menu to update other services

The chosen service is: 2
[Thu 28/05/2015 - 10:39:54.86]: The services that are restarted as a part of this operation are: VMware VirtualCenter Server, VMware VirtualCenter Management Webservices and VMware vSphere Profile-Driven Storage Service.
Enter location to the new vCenter Server SSL chain: C:\Install\ssl-certificate-updater-tool-1308332\vCenterServer-VC4002\chain.pem
Enter location to the new vCenter Server private key: C:\Install\ssl-certificate-updater-tool-1308332\vCenterServer-VC4002\rui.key
Enter vCenter Server administrator user name: domain\svc_vmware
Enter vCenter Server administrator password (will not be echoed):
"Important: Enter the password carefully. The Certificate Automation Update Tool does not check the validity of the vCenter Server database password."
"A blank or incorrect password will leave the system in an inconsistent state, which will cause the vCenter Server to become unavailable. "
"If the system becomes unstable due to a bad password, see the Troubleshooting Section of KB 2041600."
Enter the vCenter Server original database password (will not be echoed):
Enter Single Sign-On Administrator user: Administrator@vsphere.local
Enter Single Sign-On Administrator password (will not be echoed):
[.] WARNING: Certificate's `CN=VC4002.racqgroup.local, OU=vCenterServer-VC4002, O=Company, L=Location, ST=QLD, C=AU' signature uses weak one-way h
ash (SHA-1). In a secure environment it is recommended to use SHA2-256 or a stronger hash algorithm.
[.] The supplied certificate chain is valid.
Loading 'screen' into random state - done
"Restarting services... (This can take some time)"
"Stopping vCenter Web Services..."
"Stopping vCenter Server..."
"Starting vCenter Server and other services..."
[Thu 28/05/2015 - 10:45:42.32]: Last operation update vCenter Server SSL certificate completed successfully.
[Thu 28/05/2015 - 10:45:42.33]: Go to the next step in the plan that was received from Update Steps Planner.

Once you’ve had your way with vCenter, etc, you can do your ESXi hosts. The following link has info on that – Configuring CA signed certificates for ESXi 5.x hosts, and you can grab the appropriate version of Win32 OpenSSL from here. Here’s what it looks like when you use OpenSSL to generate the requests for your ESXi hosts.

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.
C:\Users\Player1>cd \
C:\>cd OpenSSL\bin
C:\OpenSSL\bin>openssl req -new -nodes -out rui.csr -keyout rui-orig.key -config
openssl.cfg
Loading 'screen' into random state - done
Generating a 2048 bit RSA private key
........+++
..........................................+++
writing new private key to 'rui-orig.key'
-----
C:\OpenSSL\bin>openssl rsa -in rui-orig.key -out rui.key
writing RSA key
C:\OpenSSL\bin>

One thing to note. I found that HA got a bit irritable until all hosts in the cluster had custom certs installed. So it’s worth turning HA off until you’re finished. If, for some reason something goes wrong wit the ESXi certs, you can re-generate the default self-signed ones with the following command:

/sbin/generate-certificates

 

Updates In some of my previous posts, I talked about a few things that I had to do to get things working. In this post, I discussed the “Missing VMware Tools ISO”. I still don’t know why the tools files were missing from the installation, but I do know that once we applied some more recent vSphere Update Manager baselines to those hosts the correct ISO files were added to the hosts.

I also covered “HP Legacy BIOS Mode and ESXi” in this post. Interestingly, you’ll need to change back to UEFI BIOS mode if you’re trying to make VirtualConnect changes to a host, as my client found out the hard way.

I also spoke about ESXi hosts and Active Directory authentication in this post. I should point out that this post by Joseph also came in handy. If you find that when you restart the services on the host it bombs out, you’ll need to manually create /var/lock/subsys. There’s a KB article from VMware that says the same thing here.

mkdir /var/lock/subsys
/etc/init.d/netlogond restart
/etc/init.d/lwiod restart
/etc/init.d/lsassd restart

And you should then be right.

EMC – VSI for VMware vSphere 6.5 Linked Mode Issue – Redux

I wrote in a previous post about having some problems with EMC’s VSI for VMware vSphere 6.5 when running in vCenter 5.5 in Linked Mode. I spoke about deploying the appliance in just one site as a workaround. Turns out that wasn’t much of a workaround. Because workaround implies that I was able to get some functionality out of the situation. While the appliance deployed okay, I couldn’t get it to recognise the deployed volumes as EMC volumes.

 

A colleague of mine had the same problem as me and a little more patience and logged a call with EMC support. Their response was “[c]urrent VSI version does not support for Linked mode, good news is recently we have several customers requesting that enhancement and Dev team are in the process of evaluating impact to their future delivery schedule. So, the linked mode may will be supported in the future. Thanks.”

 

iStock-Unfinished-Business-3

While this strikes me as non-optimal, I am hopeful, but not optimistic, that it will be fixed in a later version. My concern is that Linked Mode isn’t the problem at all, and it’s something else stupid that I’m doing. But I’m short of places I can test this at the moment. If I come across a site where we’re not using Linked Mode, I’ll be sure to fire up the appliance and run it through its paces, but for now it’s back in the box.

EMC – VSI for VMware vSphere 6.5 Linked Mode Issue

As part of a recent deployment I’ve been implementing EMC VSI for VMware vSphere Web Client v6.5 in a vSphere 5.5 environment. If you’re not familiar with this product, it “enables administrators to view, manage, and optimize storage for VMware ESX/ESXi servers and hosts and then map that storage to the hosts.” It covers a bunch of EMC products, and can be really useful in understanding where your VMs sit in relation to your EMC storage environment. It also really helps non-storage admins get going quickly in an EMC environment.

To get up and running, you:

  • Download the appliance from EMC;
  • Deploy the appliance into your environment;
  • Register the plug-in with vCenter by going to https://ApplianceIP:8443/vsi_usm/admin;
  • Register the Solutions Integration Service in the vCenter Web Client; and
  • Start adding arrays as required.

So this is all pretty straightforward. BTW the default username is admin, and the default password is ChangeMe. You’ll be prompted to change the password the first time you log in to the appliance.

 

So the problem for me arose when I went to register a second SIS appliance.

VSI1

By way of background, there are two vCenter 5.5 U2 instances running at two different data centres. I do, however, have them running in Linked Mode. And I think this is the problem. I know that you can only register one instance at a time with one vCenter. While it’s not an issue to deploy a second appliance at the second DC, every time I go to register the service in vCenter, regardless of where I’m logged in, it always points to the first vCenter instance. Which is a bit of a PITA, and not something I’d expected to be a problem. As a workaround, I’ve deployed one instance of the appliance at the primary DC and added both arrays to it to get the client up and running. And yes, I agree, if I have a site down I’m probably not going to be super focused on storage provisioning activities at my secondary DC. But I do enjoy whinging about things when they don’t work the way I expected them in the first instance.

 

I’d read in previous versions that Linked Mode wasn’t supported, but figured this was no longer an issue as it’s not mentioned in the 6.5 Product Guide. This thread on ECN seems to back up what I suspect. I’d be keen to hear if other people have run into this issue.

 

VMware – vSphere 5.5 U2 Workarounds and Random Things – Part 4

I’ve been covering a few workarounds, mishaps and random things I’ve had to do during a recent vSphere 5.5 U2 deployment. This is Part 4 in the series, and I hope some of it is useful. You can read my other posts here, here, and here.

 

Client Integration Plug-in for vSphere Web Client 

Love it or hate it, the vSphere Web Client is here to stay. If, for some reason, you’re logged into a host with credentials that you want to use to log in to your vSphere environment with, you can use pass-through authentication if you install the Client Integration Plug-in for vSphere Web Client. You can get details on how to do that here.

 

HP Legacy BIOS Mode and ESXi

This was my first time using BL460c Generation 9 blades with ESXi. While I’ve been around the block with HP blades in the past, I’ve never used them with the SD card option before. I thought this was to blame when I’d reboot the hosts and configuration items (such as persistent scratch location, syslog configuration and core dump details) would disappear. Added to this, the networking configuration on vmk0 would disappear from time to time as well. I was also getting errors such as this when applying host profiles to hosts:

“Call “HostProfileManager.GenerateConfigTaskList” for object “HostProfileManager” on vCenter Server “hostname.domain” failed.

Failed to execute command to configure or query coredump partition.”

I did some searching and chanced upon this article BL460c Gen9 + ESXi 5.5 – Special procedure when using UEFI? Seems that setting the host’s Boot Mode to Legacy BIOS Mode makes for a happier installation and on-going experience. The guy who installed the blades had set them to Legacy mode for the installation and then set them back to UEFI. I can’t tell you why this needed to occur, nor can I tell you the disadvantages of taking this approach.

 

HP_BIOS_edit

 

Windows 2012 R2 and .Net 3.5 

If you’re running your VMware applications on Windows 2012 R2, there’s a chance you’ll need to install .Net 3.5 on your guest to get things working. This is handled via Server Roles. Microsoft has a TechNet article on how to do it here. Note that you’ll need your Windows installation media, and you’ll likely need to specify an alternate source – %CDROM%\sources\sxs.

 

specify_alternate_path_dot_net_35

 

Okay, so hopefully that was useful for someone. More to follow …