jump to navigation

Upgrading to vSphere on ESXi March 22, 2010

Posted by vneil in ESXi, upgrade, VMware.

We’ve recently gone through the process of upgrading our complete VMware infrastucture to vSphere and this is a short summary of how the process went. We needed to upgrade 2 vCenter Servers, 2 Update Managers and 30 ESXi hosts servers not including any test clusters. We had been running vSphere4 on our test servers for a while and didn’t come across any problems so planned to upgrade our production environment. Luckily vSphere 4.0 Update 1 come out shortly before we were to deploy in production so after testing this on the test servers we were able to go straight to Update 1 for our production servers.
The plan was:

  • Upgrade vcenter

    which included the following :

    • stop vCenter services
    • backup vcenter db
    • backup ssl certs
    • snapshot vCenter VM
    • upgrade vCenter Server
    • Run vCenter upgrade
  • Upgrade VUM

    which included:

    • stop VUM service
    • backup VUM db
    • upgrade Update Manager
  • Upgrade converter/guided consolidation
  • Configure new licenses in vCenter
  • Configure VUM with host upgrade ISO/zip file for ESXi4.0 U1 (more on this later)
  • Backup ESXi host server configurations with vicfg-cfgbackup on the vMA server
  • Put host into maintenance mode and reboot into a network boot rescue Linux image to update Bios, nic firmware, fc firmware
  • Reboot host and remediate host with VUM to upgrade ESXi

This was mostly planned with the upgrade guide PDF and on the whole it went very well, there were a few things which I would probably add to this plan if I were to do it again.

vCenter upgrade
This went very smoothly and we didn’t come across any big problems. The only issue was with the added roles and permissions for the datastores and networks, which meant that our Windows and Linux admins couldn’t provision new disks or add networks to VMs. This was quickly resolved by clone and adding the new roles of Datastore consumer and Network consumer for each of the VM admin groups to the folders of the respective datastores and networks which were created as part of the upgrade. This is documented in the PDF (which I must have missed) on page 65.

New Roles

I really like this new feature of permissions for datastores, it means we can  have much better control with giving different datastores to different VM admins.

VUM / Hardware
Since upgrading all the hosts we have started to see various strange hardware error alarms being displayed in vCenter. The errors are things like “Host Battery Status : I2C errors” or “System Memory” errors on no particular hardware device.  Upon looking into it I thought this might have something to do with the updated CIM implementation and realised that we had loaded the standard ESXi4.0 U1 ISO image from the VMware site into VUM, maybe we should have used an IBM specific image as all our servers are the same xSeries models from IBM and came with ESXi installed on internal USB sticks. The errors we see, come and go on different servers which means it’s difficult to see if there are any real problems.

I opened a service call with IBM (who provide our VMware support) to ask if there was a customised version of ESXi for xSeries servers or if there were any known problems with ESXi4 on IBM hardware. After quite a bit of internal communication within IBM it seems there is an “IBM recovery CD” for ESXi but it doesn’t seem to come with any customisation apart from a message somewhere saying to report problems to IBM support rather than VMware support.

Now I am waiting for the CD to be sent just to see if it makes any difference while I also have open service calls with IBM for the individual problems we see. It seems the problem is to do with a “handshake problem between VMware and the IPMI” according to the service call and they are working on fixing the issue but it basically means we are stuck with these false alarms.

This is what we see:

System Memory Error image

The System Memory alarm in vCenter

Host Battery Alarm

The Host Battery Alarm in vCenter

Upgrading Firmware on ESXi

As ESXi does not have the Service Console it is harder to do things like upgrading the BIOS or changing settings on FC adapters. For this I setup our SLES Autoyast server to network boot the SLES Rescue image on the ESXi server. This allowed me to NFS mount a share with Linux versions of the BIOS update, Broadcom adapters firmware and the scli program to updated the Qlogic cards and check their settings.

Virtual Machine Upgrading
You may have noticed this missing from the upgrade plan, which is kind of true, all of the previous steps were down without any outage of applications or services (apart from vCenter but this is a controlled set of users). The next step of upgrading VM Tools and the virtual machine hardware will require further planning by the respective Windows and Linux admins (and follow in another post :-))



1. Itzik reich - March 24, 2010

you should and can download the IBM version if esx4i u1 installable fr VMware.com/go/esxi

thus version contains all the i cim providers integrated to the image

2. vneil - March 24, 2010

Thanks for the tip but I’ve spoken to IBM and they say the image on the VMware site has no customisation. They haven’t altered the CIM providers from the default version that came from VMware.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: