{"id":57,"date":"2008-08-13T09:32:20","date_gmt":"2008-08-12T23:32:20","guid":{"rendered":"https:\/\/www.penguinpunk.net\/blog\/?p=57"},"modified":"2008-08-15T09:31:04","modified_gmt":"2008-08-14T23:31:04","slug":"vmware-esxesxi-35-update-2-fail","status":"publish","type":"post","link":"https:\/\/www.penguinpunk.net\/blog\/vmware-esxesxi-35-update-2-fail\/","title":{"rendered":"VMware ESX\/ESXi 3.5 Update 2 – FAIL"},"content":{"rendered":"

This is a quick one, as I’m in a training course at the moment, but this message from VMware just hit my inbox about a serious problem with Update 2. Full details can be found here<\/a>. The text of the message is below:<\/p>\n

“Dear VMware Customers,
\nPlease find the latest update about the product expiration issue. From this point on, we\u2019ll provide an update every two hours. Thanks.<\/p>\n

Problem:
\nAn issue has been discovered by many VMware customers and partners with ESX\/ESXi 3.5 Update 2 where Virtual Machines fail to power on or VMotion successfully. This problem began to occur on August 12, 2008 for customers that had upgraded to ESX 3.5 Update 2. The problem is caused by a build timeout that was mistakenly left enabled for the release build.<\/p>\n

Affected Products:
\n\u2022\u00a0VMware ESX 3.5 Update 2 & ESXi 3.5 Update 2
\n\u2022\u00a0Reports of problems with ESX 3.5 U1 with the following 3.5 Update 2 patch applied.
\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1. ESX350-200806201-UG
\n\u2022\u00a0No other VMware products are affected.<\/p>\n

What has been done?
\n\u2022\u00a0Product and Web teams pulled the ESX 3.5 Update 2 bits from the download pages last night so no more customers will be able to download the broken build.
\n\u2022\u00a0VMware Engineering teams have isolated the cause of the problem and are working around the clock to deliver updated builds and patches for impacted customers.
\n\u2022\u00a0A Knowledgebase article has been published (http:\/\/kb.vmware.com\/kb\/1006716<\/a>), but traffic to the knowledgebase is causing time outs. A new static page has been published at http:\/\/www.vmware.com\/support\/esx35u2_supportalert.html<\/a> that customers and partners will be able to view.
\n\u2022\u00a0The phone system has been updated to advise customers of the problem
\n\u2022\u00a0Vmware partners have been notified of the issue.<\/p>\n

Workarounds:
\n1.\u00a0Do not install ESX 3.5 U2 if it has been downloaded from VMware\u2019s website or elsewhere prior to August 12, 2008.
\n2.\u00a0Set the host time to a date prior to August 12, 2008. This workaround has a number of very serious side affects that could impact product environments. Any Virtual Machines that sync time with the ESX host and serve time sensitive applications would be broken. These include, but are not limited to database servers, mail servers, & domain administration systems.
\nNext Steps:
\nVMware to notify customers who have downloaded this version and provide an update every two hours.
\nResolution:
\nVMware Engineering has isolated the root cause and is working to produce an express patch for impacted customers today. The target timeframe is 6pm, August 12, 2008 PST.<\/p>\n

FAQ:
\n1.\u00a0What would this express patch do?<\/p>\n

More information will be provided in subsequent communication updates.
\n2.\u00a0Will VMware still reissue the upgrade media and patch bundles in the timeframe that has been communicated?<\/p>\n

Yes. We still plan to reissue upgrade media by 6pm, August 13 PST (instead of noon, August 13 PST) and all update patch bundles later in the week. We will provide an ETA for the update patch bundles subsequently. NOTE: the “patch bundles” referred to here are for the patches listed above under “Affected Products” and the other bundles released at GA. They are not the same as the express patch which is targeted for 6pm, August 12, 2008 PST as stated above.
\n3.\u00a0Why does VMware plan to reissue the upgrade media before the patch bundles? That is a wrong priority call!<\/p>\n

This is not a matter of priority. Since we can get done building and testing the upgrade media before the patch bundles, we want to make that available to customers first instead of reissuing all the binaries later in the week.
\n4.\u00a0Can VMware issue a patch that opens the licensing backdoor in the next hour as a critical measure?<\/p>\n

There is no licensing backdoor in our code.
\n5.\u00a0Does this issue affect VC 2.5 Update 2?<\/p>\n

No.
\n6.\u00a0What is VMware doing to make sure that the problem won\u2019t happen again?<\/p>\n

We are making improvements on all fronts. The product team had endeavored to deliver a release with support customers deem important. But we fell short and we are deeply sorry about all the disruption and inconveniences we have caused. We have identified where the holes are and they will be addressed to restore customers\u2019 confidence. ”<\/p>\n

I’m thinking that next week when I’m back in town might be a little busy with this one …<\/p>\n

**Update 1** – 13\/08\/2008 – 14:15 (AEST)<\/p>\n

VMware have released an express update for this issue. Please look here<\/a>\u00a0for more information concerning the patch and its implementation. You can use VUM or run esxupdate on the affected hosts to install the patch. Which is nicer than frigging about with times on your ESX nodes. I haven’t had time to run the patch, but I’m sure I’ll hear about it from my customers if it is problematic :)<\/p>\n

**Update 2** – 13\/08\/2008 – 15:30 (AEST)<\/p>\n

I\u00a0 may have spoken too soon regarding express patch deployment – according to the latest missive from VMware, the express patches have only been tested using esxupdate, not VUM.<\/p>\n

“There are two express patches: one for ESX 3.5 Update 2 and one for ESXi 3.5 Update 2. They are specifically targeted for customers who have installed or fully upgraded to ESX\/ESXi 3.5 Update 2 or who have applied the ESX350-200806201-UG patch to ESX\/ESXi 3.5 or ESX\/ESX 3.5 Update 1 hosts. For customers who haven\u2019t done either, these express patches should not be applied.
\nTo be noted is that these patches have been validated to work with esxupdate. However, testing with the VMware Update Manager is still under way. In subsequent communications, we will provide confirmation whether the patches work with VMware Update Manger or if a re-spin is required.
\nWe are currently testing an option to apply the patch without requiring VMotion or VM power-off and re-power-on at the point of patch application. To immediately refresh vmx on the VM, one can VMotion off running VMs, apply the patches and VMotion the VMs back. If VMotion capability is not available, VMs can be powered off before the patches are applied and powered back on afterwards.”<\/p>\n

You having fun yet?<\/p>\n

**Update 3** – 15\/08\/2008 – 09:30 (AEST)<\/p>\n

Apparently VMware have now released updated install media for ESX 3.5 Update 2 and ESX3i, including upgrade files, etc. So go to http:\/\/www.vmware.com\/download<\/a> and get yourself some new media.<\/p>\n","protected":false},"excerpt":{"rendered":"

This is a quick one, as I’m in a training course at the moment, but this message from VMware just hit my inbox about a serious problem with Update 2. Full details can be found here. The text of the message is below: “Dear VMware Customers, Please find the latest update about the product expiration […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7,4],"tags":[9,51,52,786],"class_list":["post-57","post","type-post","status-publish","format-standard","hentry","category-humour","category-vmware","tag-esx","tag-fail","tag-update2","tag-vmware"],"_links":{"self":[{"href":"https:\/\/www.penguinpunk.net\/blog\/wp-json\/wp\/v2\/posts\/57","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.penguinpunk.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.penguinpunk.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.penguinpunk.net\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.penguinpunk.net\/blog\/wp-json\/wp\/v2\/comments?post=57"}],"version-history":[{"count":7,"href":"https:\/\/www.penguinpunk.net\/blog\/wp-json\/wp\/v2\/posts\/57\/revisions"}],"predecessor-version":[{"id":59,"href":"https:\/\/www.penguinpunk.net\/blog\/wp-json\/wp\/v2\/posts\/57\/revisions\/59"}],"wp:attachment":[{"href":"https:\/\/www.penguinpunk.net\/blog\/wp-json\/wp\/v2\/media?parent=57"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.penguinpunk.net\/blog\/wp-json\/wp\/v2\/categories?post=57"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.penguinpunk.net\/blog\/wp-json\/wp\/v2\/tags?post=57"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}