How To Update vCenter Server Appliance 6.5

VMware has released vCenter 6.5.0d which allowes me to update vCSA (vCenter Server Appliance) in our lab environment and to document its steps. The update process has been greatly simplified with the Appliance Management tool which is also used to manage the appliance’s configuration or parts of it.

This guide will take you through the steps to update your vCSA with three different methods using the Appliance Management tool.

Take Backup

Before trying out any of these methods, make sure to snapshot and/or take a backup of the appliance just in case the update fails. Throughout this guide, I successfully reverted back to snapshot a number of times not because any of the update methods failed but because I had to make sure that each of the methods covered here worked flawlessly. So, do yourself a favor and snapshot your appliance more so if it’s being used for production.

Taking a snapshot of vCSA

Updating vCSA Over the Internet 

This method works only if the appliance has Internet connectivity (ports 80,443) which it needs to connect to VMware’s online repositories.

To begin with, point your browser to http://<vCSA IP Address or hostname>:5480 and log in as root. Remember that we’re accessing the appliance itself and not the vCenter Server component so user accounts like administrator@vsphere.local will not work.

The Appliance Management tool used to configure and update vCSA

After you log in, click on Update and hit the Settings button as shown in following image.

Update settings page

From the Update Settings dialog, you can either use the default VMware repository URL or specify one yourself. The default repository is automatically points to the latest appliance update as shown in Fig. 4. You can also schedule the time at which the vCSA polls for updates. This, however, does not imply that updates are automatically downloaded.

Link to the latest available vCSA update is set automatically

If the Check for updates automatically option is ticked off, click on Check Updates and select Check Repository. This will poll the VMware repository for the latest available update and display the link as such. Expanding More Details, will reveal more information on the update being installed along with a link to the KB article describing it. When required, use this link to correlate the current version with that being updated.

Manually checking for the latest update. Full details for update are also displayed.

Updating the appliance, is a simple matter of clicking on Install Updates and selecting Install All Updates as per following screenshot.

Ready to update vCSA

The upgrade process may take a while and at times you may think it’s actually stuck. Just remember that the process needs to download 1.5GB worth of file which can take time depending on the environment, available bandwidth and what not. In my case – nested environment on a heavily used ESXi host – , 40 minutes into the update and the installer was still stuck at 40%.

To verify that the vCSA was actually doing something, I logged on the vCenter Server hosting it using the vSphere Web Client. I then checked the vCSA VM’s network performance graph for any signs of activity. As shown in Fig. 7,  the VM was pretty busy on the networking front with activity spiking immediately as soon as the update kicked in. 

Network activity on the vCSA’s VM indicating that the update is being downloaded

At one point, the installer lost connectivity to the appliance – probably because it timed out – so I wasn’t quite sure if the update completed successfully. If this happens, SSH to the vCSA and check the contents of a log file called software-packages.log which you’ll find under /storage/log/vmware/applmgmt. Run the following command:

cat /storage/log/vmware/applmgmt/software-packages.log | grep 'Packages upgraded successfully'

Checking the software-packages log file to determine the result of the update

If the update completed successfully, you should see a line containing Packages upgraded successfully, reboot is required.  If that’s the case, proceed with rebooting the appliance so the changes can take root. If not, scour the log file for hints as to what went wrong and revert back to snapshot. This is all shown in the video below. The appliance’s version will read to the latest even though the appliance needs to be rebooted. You can verify this from the Appliance Management tool and the vCSA’s console.

Updating vCSA Locally

If Internet access is denied to the appliance, which is pretty normal in production environments, you’ll need to download the update manually and use one of the two methods described next.

The ISO Method

You need to download the vCSA update – as an ISO file – from https://my.vmware.com/group/vmware/patch#search. To download the update, you need to sign up for a VMware account unless you already have. Once you do that, select VC followed by the latest version from the drop-down menus as shown in Fig.9, marked 1 and 2. Then, select the latest available update by ticking the box next to it (3) and click on Download (4).

Downloading the vCSA update as an ISO image

As shown in Fig. 10, I’ve uploaded the ISO file to a datastore on the ESXi host where the vCSA VM resides. I then mounted it as a CD/DVD drive from the vCSA VM’s settings. The same update process is repeated using the Appliance Management tool.

Select the Check CDROM update option to verify the ISO image, the details for which are displayed on the Update screen. If all’s well and good, select Install CDROM Updates (4) to run the update process.

Updating the appliance using a locally mounted ISO image

Updating from ISO, reduces the time taken to update the appliance mainly because there are no update files to download.

The update process in progress

This time round, there were no time-outs and the appliance update took only 5 minutes to successfully complete. The appliance is then rebooted by clicking OK – which brings up another dialog box prompting for a reboot – or via the Reboot button on the Summary page.

A successful update! An appliance reboot is mandatory.

The Web Server Method

If complicating matters is your thing, here’s an alternative method you can try. This time, you need to download the vCSA update bundle which is a zipped archive. This, pretty much, contains the same RPM packages comprising the ISO file save for a couple of manifest files. The zip file is downloadable from my.vmware.com from the VMware vCenter Server 6.5.0d downloads section. 

The vCSA update bundle downloadable from my.vmware.com

The bundle is then extracted to the root folder of a web server. For this example, I’ve used IIS. I extracted the archive’s contents to c:\inetpub as shown in Fig. 14 and modified the Default Web Site to point directly to it. Directory Browsing must be enabled for Default Website unless the setting is inherited.

Extracting the update bundle to an IIS server and setting the default website to point to it

Enabling directory browsing on IIS

Lastly, you need to create MIME types for the .sign and .json files which the vCSA reads from the files present under the two folders – package-pool and manifest – extracted to c:\inetpub.

Manifest files from the extracted update bundle

To add new MIME types, just click on the MIME Types icon and add them via the Add link at the top-right corner or by right-clicking on the MIME Types page in IIS. Add the 2 new mime types as shown in Figure 17.

Note: This was tested on IIS 8 running on Windows Server 2012. Additional MIME types, perhaps for the RPM packages, may need to be created.

Creating MIME types in IIS 8

That’s all there is to it as far as IIS configuration is concerned. For authentication, I used anonymous which is enabled by default. You may wish to use other forms of authentication if security is a concern.

To update vCSA from the IIS repository, click on Updates, Settings and type in the IP address of the IIS server as shown below. Optionally, add a username and password if you set up any other form of authentication other than anonymous.

Point the vCSA to download the update bundle from the IIS server

The appliance will upgrade identically to the previous methods used. On completion, you are once again asked to reboot.

Rebooting the appliance after updating

Troubleshooting

You may come across a Download Failed error message when testing this method for the first time. This generally occurs due to 401 or 404 errors on the Web Server’s side. While testing this method, I took hints from the software-packages.log on the vCSA to iron out any problems I ran into. For instance, 404 errors told me that I needed to create mime types for the .sign and .json files since IIS, by default, doesn’t know about them which leads to a file not found error.

Below is an example of what you’ll see in the log file. The entries point to something wrong with how authentication is set up on IIS; I intentionally disabled anonymous authentication to reproduce the error.

2017-04-27T07:05:57.117 [11235]DEBUG:vmware.vherd.base.software_update:Extracting installed package information
2017-04-27T07:05:57.117 [11235]DEBUG:vmware.vherd.base.software_update:/storage/core/software-packages directory created successfully
2017-04-27T07:05:57.117 [11235]DEBUG:vmware.vherd.base.software_update:WGET: http://192.168.16.71/manifest/manifest-latest.xml
2017-04-27T07:05:57.117 [11235]DEBUG:vmware.vherd.base.software_update:Failure:
out=
error=--2017-04-27 07:05:57-- http://192.168.16.71/manifest/manifest-latest.xml
Connecting to 192.168.16.71:80... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Username/Password Authentication Failed.

Conclusion 

Updating vCenter Server Appliance is pretty straightforward, something you can do using any of the three methods outlined in this guide. Regardless of the method used, it is always important to take a backup of the vCSA, and any other critical component for that matter, before updating or upgrading. There’s no guarantee that an update will succeed, so at the risk of repeating myself, being able to recover from a failed update or upgrade is paramount to business continuity.