We had several vSphere vCenter 4.0U2 servers running on Windows 2003 R2 32 bit virtual machines and seeing as we needed to move to 64bit for 4.1 I took the opportunity to  switch to Windows 2008 R2 64bit. A test vCenter server has a MS-SQL Express database on the same server and two production vCenter servers (in Linked mode) have databases on separate MS-SQL 2005 servers.

We built new Windows 2008R2 64bit servers in preparation and I used the datamigration tool from the vCenter distribution to perform the migration of data and install of 4.1.

Issues

MS-SQL Express database with data migration

This occurred with the MS-SQL Express migration, the datamigration tool did not backup the data as expected but also did not log any error messages during the backup procedure. The install runs fine but just creates a new empty database. This is due to a MS-SQL Express database that has been upgraded and registry settings being set wrong. It is detailed in this KB article (1024380) with the fix.

This is from the backup .log showing a good backup, if it says DB type of Custom it will not work:

[backup]  Backing up vCenter Server DB...
[backup]  Checking vCenter Server DB configuration...
[backup]  vCenter Server DB type: bundled

Running as Administrator

As a non-Windows bod, one thing that keeps catching me out on Windows 2008R2 is that to do most things, you need to explicitly say to run them as an Administrator even if your userid is a local administrator. For example, when running the install.cmd from the datamigration tool, it will run as your normal userid but will fail fairly soon. Right click on the install.cmd and choose “Run as Adminstrator” for a simple life.

MS-SQL Express – transaction logging from bulk to simple

Another issue with the MS-SQL Express upgrade path was with the database transaction log filling up after a few days. Again, not being that au fait with MS-SQL management I found the solution to this one here in a post by Patters. It seems the upgrade process sometimes leaves the database logging mode as ‘bulk-logged‘, using the Management Studio Express tool I switched it to ‘Simple‘.

MS-SQL 2005 Server databases

I also had a similar problem with just one of the production databases on the MS-SQL 2005 server which was left in “Bulk-logged” mode after the upgrade and the DBA had to switch it back to Simple.

Sysprep files – all in one directory

As the server was a new install I also needed to repopulate the Sysprep files, following the instructions  in the KB article http://kb.vmware.com/kb/1005593 . One thing I missed was to expand the deploy.cab file from the downloaded file, what I ended up doing was to make sure there were no sub directories under the relevant sysprep directory (srv2003 in this case) and put all the files from downloaded file and the files from deploy.cab in that directory.

So far we’ve done a few host upgrades of test ESXi hosts to 4.1 without problems, production yet to come.

I had setup a couple of test virtual machines in this scenario, the first I setup with a physical RDM of the shared LUN and enabled SCSI bus sharing on the adapter. Then for the second test virtual machine I pointed it at the mapped VMDK file of the physical RDM from the first virtual machine, this seemed to be obvious way to setup sharing of the physical RDM between VMs. This all works and both Linux servers can use the LUN and the clustered filesystem. The problem I came across is that as soon as you enable SCSI bus sharing vMotion is disabled. I wanted the ability to migrate these virtual machines if I needed to so I looked into how to achieve this.

Adding config.vpxd.filter.rdmFilter in vCenter

If you only have one virtual machine and do not enable the SCSI bus sharing then vMotion is possible.  The solution I came up with was to setup each virtual machine the same, ie. each to have the LUN as a physical RDM but of course once you allocated the LUN to one virtual machine it is not visible to any other VM.  The easiest way to allow this to happen is with the config.vpxd.filter.rdmFilter configuration option, this is set in the vCenter settings under Administration > vCenter Server Settings > Advanced Settings and is detailed in this KB article. If the setting is not in the list of Advanced Settings it can be added.

As Duncan Epping rightly describes here it’s not really wise to leave this option set to false. What I did was to set the option to false, carefully allocate the LUN to the three Linux servers I was setting up and then set it back to true.

With the virtual machines setup like this the restriction is that they cannot run on the same host so I set anti-affinity rules to keep them apart.

It is also possible to do with via the command line without the need to set the Advanced Setting as described in this mail archive I found. I haven’t tried this method.

I don’t know if this setup is supported or not but it seems to achieve the goal we need. I’d be interested to hear any opinions on this.

We started running SLES 10 on a ESXi 3.5 cluster and for the VM tools we started using the RPMs packaged from VMware. This was fine until we upgraded to ESXi 4.0 on the hosts, after that VM tools that are installed from the RPMs show up in vCenter with a status of “unmanaged“. This is as described in the documentation. We decided then to switch to using the VM tools installed from the ESXi hosts and VUM, this meant that the status returned to “OK” in vCenter and we had the benefit of being able to upgrade the virtual machines’s VMtools with VUM.

The problem started when we started upgrading our Linux servers from SLES 10 SP3 to SLES 11 SP1, it seems the VM tools distributed with ESXi and downloaded via VUM do not distribute binary kernel modules for SP1 , they have SLES 11 GA but not SP1. This is fine if you have SLES11 SP1 virtual machines with compilers and kernel sources but our production servers do not have these for obvious reasons. The upshot of this is that we have switched back to the RPM distributed tools and will have to put up with the status saying “unmanaged“.

I have checked the VM tools that come on the ESXi4.1 installation and these also do not have the binary modules for SP1.

This is quite annoying as we had a small battle with the other Unix admins to switch to using VUM to upgrade VM tools and to now have to switch back to RPMs means it will be almost impossible to ever switch back to using VUM as and when SLES11 SP1 is properly supported with VMware ESXi.

I signed up and downloaded the demo installer of Hyperic 4.4  which included the server and agent for Linux. I installed the in a standard SLES server and started the server. This was all quite straight forward and it also installed a copy of PostgreSQL to use as it’s database. When started the server it was accessible from a browser on port 7080. I signed in with the default user of hqadmin.

Initial screen after install

Starting dashboard

As with most monitoring software it needs agents. I installed an agent on the Windows server running VMware vSphere Server by downloading the win32 agent from the website and unpacking it on the W2003 server

Then it’s just a case of running hq-agent.bat install and  hq-agent.bat start

Install and start of Windows HQ agent

Answer the questions, I took the defaults. When it starts it goes off and registers itself with the HQ server using the details given.

The agent should discover application details from the  system it is on and appear after a refresh in the Auto-Discovery pane on the Dashboard of the HQ server. It should show VMware vCenter,  select Add to Inventory.

HQ agent auto-discovery

Unfortunately there seems to be a problem with the Windows HQ agent and vSphere as mentioned here

You need to edit the agent.properties file in the conf directory of the HQ agent with the details you gave when it was installed. Then restart the agent with hq-agent restart
agent.setup.camIP=10.14.0.35
agent.setup.camPort=7080
agent.setup.camSSLPort=7443
agent.setup.camSecure=yes
agent.setup.camLogin=hqadmin
agent.setup.camPword=hqadmin
agent.setup.agentIP=10.14.0.79
agent.setup.agentPort=2144
agent.setup.resetupTokens=no
#
agent.setup.unidirectional=yes

To allow the agent to discover vSphere details, select the Resources tab -> windows server -> hostname VMware vCenter -> Inventory tab. Edit the Configurations Properties (near the bottom of the page) add a username and password that can access the vCenter sdk. After a few minutes a refresh of the Dashboard should start to show discovered ESX/ESXi hosts and VMs  in the Recently Added pane.

The ESXi hosts will show up under the Resources tab under Platforms, along with your agents. Under Resources -> HQ VSphere you get a topology view of vCenter/Host/VM

There seems to be a lot of options for monitoring and alerting which I haven’t really looked into yet but it looks quite promising.

A big issue is that it doesn’t seem all that stable at the moment (it could be because this was a quick demo install and I haven’t really tuned it at all) for example once or twice I deleted one agent in the Resources tab and several resources disappeared. If an agent disappears from view it seems one way to get it back is to stop the agent and then delete the contents of the data directory and then when the agent is restarted (with hq-agent.bat start) it then registers again on the server.

The next problem I encountered was adding a second vSphere server, the view under Resources->HQ vSphere got very confused. First off both vCenter servers were listed but the host servers were under  the wrong vCenter server , then one of the vCenter servers disappeared completely.

I’ll continue to test it and search forums to see if there are any solutions to these problems.

If agents are installed in the running VMs as well it seems that you can drill down from your running vSphere server, to host server, to VM and then down to applications running in that VM. This could prove quite useful in problem solving any particular issues with performance.

# mpstat 5 5
Linux 2.6.16.60-0.21-smp (xxxx) 	05/31/10
CPU   %user   %nice    %sys   %steal   %idle    intr/s
all    0.00    0.00    0.60    0.00   99.40   154596.40

A lot of interrupts (intr/s) for an idle system, compare this to a ‘normal’ busy system:

$ mpstat 5
Linux 2.6.16.60-0.21-smp (xxxx) 	06/03/2010
CPU   %user   %nice    %sys   %steal   %idle    intr/s
all   81.24    0.00   18.36    0.00    0.00     9832.73

We have around 120 Linux virtual machines and I was only seeing this on about 5 systems. Looking at the output from procinfo it easy to see the interrupts are timer interrupts:

# procinfo
Linux 2.6.16.60-0.21-smp (geeko@buildhost) (gcc 4.1.2 20070115) #1 SMP Tue May 6 12:41:02 UTC 2008 1CPU [xxxxx]

Memory:      Total        Used        Free      Shared     Buffers
Mem:       3867772      494576     3373196           0       11228
Swap:      4194296           0     4194296

Bootup: Mon May 31 14:24:38 2010    Load average: 0.00 0.16 0.18 2/152 5427

user  :       0:00:19.94   2.8%  page in :    1054996  disk 1:    26937r    7117w
nice  :       0:00:05.30   0.7%  page out:      28856  disk 2:      124r       0w
system:       0:00:20.78   2.9%  page act:      30744
IOwait:       0:01:28.12  12.2%  page dea:          0
hw irq:       0:00:01.38   0.2%  page flt:    1514598
sw irq:       0:00:01.32   0.2%  swap in :          0
idle  :       0:09:32.81  79.6%  swap out:          0
uptime:       0:11:59.66         context :     297632

irq  0: 101039309 timer                 irq  9:         0 acpi
irq  1:         9 i8042                 irq 12:       114 i8042
irq  3:         1                       irq 14:      5660 ide0
irq  4:         1                       irq169:     30998 ioc0
irq  6:         5                       irq177:      1113 eth0
irq  8:         0 rtc                   irq185:      2480 eth1 

I picked one and just rebooted it from the command line in Linux. After the reboot it still showed the same symptoms, a very high intr/s. Shutting down all services and daemons still didn’t help. I then shutdown and powered off the VM and then after the power on it was ok, the high interrupts per second had stopped.

We then experimented with trying a vmotion with one of these systems which had a high intr/s, this also seemed to work. “Good!” we thought a non-interruptive fix but when we tried with vmotion on another server it didn’t fix the problem. Then trying a vmotion again on the same server fixed it ?!

It seems that there is no pattern in fixing this, a cold reboot works or multiple vmotions sometimes helps. All these systems were SLES servers with SP2 running the SMP Linux kernel even though most only had one vCPU (but not all).

Definitely a strange one. I’ll keep an eye on this.

This meant doing things like VLAN tagging / trunking was not feasible, we had to provide connections to 4 different virtual machine networks so we had a lot of network connections and did not have redundancy on all. The servers had 8 network connections but to allow for ESXi / vMotion networks and the VM networks, the maintenance networks for the VMs did not get redundancy plus we had to drop one production network which wasn’t required straight away.

Later we organised an in-house VMware training (given by the excellent Eric Sloof) and invited a couple of the key guys from the network team who we had been working with. This had the great affect of allowing them to see how VMware handled it’s internal network and gave us all a chance to brainstorm ways of changing the network connections as we had also started getting pressure to provide connections to the mainframe production network we had left out.

The result was a much better architecture and reorganisation of how the adapters were used by VMware and incorporatation of 802.1Q VLAN tagging, proving the benefit of spreading the word and getting buy-in from other departments you deal with.

Seeing as I had 20 ESXi servers in the cluster to change I created a script to do the definition for me. This wasn’t my own work but cobbled together from scripts from Powershell heroes like Alan Renouf and Luc Dekens.

Here is the script to create everything except the Management Network:

$esxserver= Get-VMHost esxserver4.vmware.in
$vmoip = "172.1.101.4"

# Add vmnic4,vmnci6 to vSwitch0
$vSwitch0 = get-virtualswitch -vmhost $esxserver -name vSwitch0
Set-VirtualSwitch -VirtualSwitch $vSwitch0 -Nic vmnic4,vmnic6 -NumPorts 128

# add serverfarm portgroup to vSwitch0
New-VirtualPortGroup -Name "serverfarm" -VirtualSwitch $vSwitch0

# Configure portgroup policies for vSwitch0
$hostview = $esxserver | Get-View
$ns = Get-View -Id $hostview.ConfigManager.NetworkSystem

# set failover policy for serverfarm portgroup
$pgspec = New-Object VMware.Vim.HostPortGroupSpec
$pgspec.vswitchName = "vSwitch0"
$pgspec.Name = "serverfarm"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy
# create object for nic teaming in port group
$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy
$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy
$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic4","vmnic6")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic1")
# load balancing
$pgspec.policy.NicTeaming.policy = "loadbalance_srcid"
# link failure
$pgspec.policy.NicTeaming.failureCriteria = New-Object vmware.vim.HostNicFailureCriteria
$pgspec.policy.NicTeaming.failureCriteria.checkBeacon = $false
# Failback
$pgspec.policy.NicTeaming.RollingOrder = $false
# notify switches
$pgspec.policy.NicTeaming.notifySwitches = $true
$ns.UpdatePortGroup($pgspec.Name,$pgspec)

# set failover policy for Management Network portgroup
$pgspec = New-Object VMware.Vim.HostPortGroupSpec
$pgspec.vswitchName = "vSwitch0"
$pgspec.Name = "Management Network"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy
# create object for nic teaming in port group  (opposite failover to other port group)
$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy
$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy
$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic1")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic4","vmnic6")
# load balancing
$pgspec.policy.NicTeaming.policy = "failover_explicit"
# link failure
$pgspec.policy.NicTeaming.failureCriteria = New-Object vmware.vim.HostNicFailureCriteria
$pgspec.policy.NicTeaming.failureCriteria.checkBeacon = $false
# Failback
$pgspec.policy.NicTeaming.RollingOrder = $false
# notify switches
$pgspec.policy.NicTeaming.notifySwitches = $true
$ns.UpdatePortGroup($pgspec.Name,$pgspec)

# create vSwitch1 for mainframe
New-VirtualSwitch -VMhost $esxserver -Nic vmnic3,vmnic5 -NumPorts 128 -Name "vSwitch1"
$vSwitch1= Get-VirtualSwitch -vmhost $esxserver -Name "vSwitch1"
New-VirtualPortGroup -Name "mainframe" -VirtualSwitch $vSwitch1

# create vSwitch2 for server mgmt and vmotion
New-VirtualSwitch -VMhost $esxserver -Nic vmnic0,vmnic2 -NumPorts 128 -Name "vSwitch2"
$vSwitch1= Get-VirtualSwitch -vmhost $esxserver -Name "vSwitch2"

# create servermgmt_linux (vlan3001)
New-VirtualPortGroup -Name "servermgmt_linux" -VLanId 3001  -VirtualSwitch $vSwitch2

# create servermgmt_win (vlan3002)
New-VirtualPortGroup -Name "servermgmt_win" -VLanId 3002  -VirtualSwitch $vSwitch2

# create vmotion (vlan3099)
New-VMHostNetworkAdapter -VMHost $esxserver -PortGroup vmotion -VirtualSwitch $vSwitch2 -IP $vmoip -SubnetMask 255.255.255.0 -VMotionEnabled $true -EA Stop

$hostview = $esxserver | Get-View
$ns = Get-View -Id $hostview.ConfigManager.NetworkSystem

# set failover policy for vmotion portgroup
$pgspec = New-Object VMware.Vim.HostPortGroupSpec
$pgspec.vswitchName = "vSwitch2"
$pgspec.Name = "vmotion"
$pgspec.vlanId = "3099"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy
# create object for nic teaming in port group
$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy
$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy
$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic2")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic0")
# load balancing
$pgspec.policy.NicTeaming.policy = "failover_explicit"
# link failure
$pgspec.policy.NicTeaming.failureCriteria = New-Object vmware.vim.HostNicFailureCriteria
$pgspec.policy.NicTeaming.failureCriteria.checkBeacon = $false
# Failback
$pgspec.policy.NicTeaming.RollingOrder = $false
# notify switches
$pgspec.policy.NicTeaming.notifySwitches = $true
$ns.UpdatePortGroup($pgspec.Name,$pgspec)

# set failover policy for servermgmt_linux portgroup
$pgspec = New-Object VMware.Vim.HostPortGroupSpec
$pgspec.vswitchName = "vSwitch2"
$pgspec.Name = "servermgmt_linux"
$pgspec.vlanId = "3001"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy
# create object for nic teaming in port group
$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy
$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy
$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic0")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic2")
# load balancing
$pgspec.policy.NicTeaming.policy = "failover_explicit"
# link failure
$pgspec.policy.NicTeaming.failureCriteria = New-Object vmware.vim.HostNicFailureCriteria
$pgspec.policy.NicTeaming.failureCriteria.checkBeacon = $false
# Failback
$pgspec.policy.NicTeaming.RollingOrder = $false
# notify switches
$pgspec.policy.NicTeaming.notifySwitches = $true
$ns.UpdatePortGroup($pgspec.Name,$pgspec)

# set failover policy for servermgmt_win portgroup
$pgspec = New-Object VMware.Vim.HostPortGroupSpec
$pgspec.vswitchName = "vSwitch2"
$pgspec.Name = "servermgmt_win"
$pgspec.vlanId = "3002"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy
# create object for nic teaming in port group
$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy
$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy
$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic0")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic2")
# load balancing
$pgspec.policy.NicTeaming.policy = "failover_explicit"
# link failure
$pgspec.policy.NicTeaming.failureCriteria = New-Object vmware.vim.HostNicFailureCriteria
$pgspec.policy.NicTeaming.failureCriteria.checkBeacon = $false
# Failback
$pgspec.policy.NicTeaming.RollingOrder = $false
# notify switches
$pgspec.policy.NicTeaming.notifySwitches = $true
$ns.UpdatePortGroup($pgspec.Name,$pgspec)

• Upgrade vcenter

  which included the following :

  • stop vCenter services
  • backup vcenter db
  • backup ssl certs
  • snapshot vCenter VM
  • upgrade vCenter Server
  • Run vCenter upgrade
• Upgrade VUM

  which included:

  • stop VUM service
  • backup VUM db
  • upgrade Update Manager
• Upgrade converter/guided consolidation
• Configure new licenses in vCenter
• Configure VUM with host upgrade ISO/zip file for ESXi4.0 U1 (more on this later)
• Backup ESXi host server configurations with vicfg-cfgbackup on the vMA server
• Put host into maintenance mode and reboot into a network boot rescue Linux image to update Bios, nic firmware, fc firmware
• Reboot host and remediate host with VUM to upgrade ESXi

This was mostly planned with the upgrade guide PDF and on the whole it went very well, there were a few things which I would probably add to this plan if I were to do it again.

vCenter upgrade
This went very smoothly and we didn’t come across any big problems. The only issue was with the added roles and permissions for the datastores and networks, which meant that our Windows and Linux admins couldn’t provision new disks or add networks to VMs. This was quickly resolved by clone and adding the new roles of Datastore consumer and Network consumer for each of the VM admin groups to the folders of the respective datastores and networks which were created as part of the upgrade. This is documented in the PDF (which I must have missed) on page 65.

I really like this new feature of permissions for datastores, it means we can  have much better control with giving different datastores to different VM admins.

VUM / Hardware
Since upgrading all the hosts we have started to see various strange hardware error alarms being displayed in vCenter. The errors are things like “Host Battery Status : I2C errors” or “System Memory” errors on no particular hardware device.  Upon looking into it I thought this might have something to do with the updated CIM implementation and realised that we had loaded the standard ESXi4.0 U1 ISO image from the VMware site into VUM, maybe we should have used an IBM specific image as all our servers are the same xSeries models from IBM and came with ESXi installed on internal USB sticks. The errors we see, come and go on different servers which means it’s difficult to see if there are any real problems.

I opened a service call with IBM (who provide our VMware support) to ask if there was a customised version of ESXi for xSeries servers or if there were any known problems with ESXi4 on IBM hardware. After quite a bit of internal communication within IBM it seems there is an “IBM recovery CD” for ESXi but it doesn’t seem to come with any customisation apart from a message somewhere saying to report problems to IBM support rather than VMware support.

Now I am waiting for the CD to be sent just to see if it makes any difference while I also have open service calls with IBM for the individual problems we see. It seems the problem is to do with a “handshake problem between VMware and the IPMI” according to the service call and they are working on fixing the issue but it basically means we are stuck with these false alarms.

This is what we see:

The System Memory alarm in vCenter

The Host Battery Alarm in vCenter

Upgrading Firmware on ESXi

As ESXi does not have the Service Console it is harder to do things like upgrading the BIOS or changing settings on FC adapters. For this I setup our SLES Autoyast server to network boot the SLES Rescue image on the ESXi server. This allowed me to NFS mount a share with Linux versions of the BIOS update, Broadcom adapters firmware and the scli program to updated the Qlogic cards and check their settings.

Virtual Machine Upgrading
You may have noticed this missing from the upgrade plan, which is kind of true, all of the previous steps were down without any outage of applications or services (apart from vCenter but this is a controlled set of users). The next step of upgrading VM Tools and the virtual machine hardware will require further planning by the respective Windows and Linux admins (and follow in another post )

One thing that was mentioned by the main guest VMware’s Charu, which I haven’t tried before, was to setup persistent logging for an ESXi host. As he mentioned, although there is logging in ESXi which gets stored in the /var/log/ directory it is all in memory so if the host crashes, all logs are lost. While I already had the remote syslog setup it seems this local logging is a different log file.

On one of our test servers I went into Configuration->Advanced Settings (Software) -> Syslog -> Local and entered a directory and filename on a VMFS datastore and it started logging immediately to the specified file.

I then tried to see if it was a copy of a log that was already being created somewhere on the host and while it looks similar to the /var/log/vmware/hostd.log file it seems to be more verbose.

The other logs that I am aware of that are useful are:
/var/log/messages
/var/log/vmware/hostd.log
/var/log/vmware/vpx/vpxa.log

These are available via the browser interface (http://hostname/host). The messages file is what is sent via to the remote syslog if configured. There are also logs in /var/log/vmware/aam/that relate to the HA agent on the host and if there problems with a host enabling the agent it is sometimes fruitful to look in here at the latest files.

I would say that setting up both local persistent and remote syslog logging as a very good recommendation in configuring ESXi hosts especially in a production environment.

It looks like if it was mentioned at PEX it might be coming soon, I had kind of hoped it might have been in vSphere as the next logical extension of DRS after CPU and Memory resource scheduling.

The way we run our current cluster  is one big cluster with test and production VMs spread across all the host servers with their VMDKs also mixed on datastores. They are distributed into appropriate  resource groups and all this helps to even the workload and make full use of all resources.

The cluster is not heavily burdened at the moment and we are continuing to P2V Windows servers and deploy new Linux servers onto it. I see IO DRS as an extremely helpful tool with the growth of the number of VMs in the cluster to keep the necessary prioritising of production over testing.

In the meantime I’ll keep an eye out for any more news on IO DRS.

To make this simple when setting up the cluster we went for a standard 500GB size for the LUNs and had about 8 LUNs allocated to start with. We needed to somehow level the usage of these LUNs across the 4 paths.

These servers were running ESXi 3.5 and the round robin path policy was still experimental this left a decision of either Fixed or Most Recently Used (MRU) as the multipath policy to use. After some investigation I chose to use Fixed multipath as with MRU you could possibly end up with all LUNs on one path if several path failures go unnoticed.

To simplify the spreading of LUNs over the multiple paths the LUNs were each allocated a path in order, for example LUN 1 to path 1 , LUN 2 to path 2 ..etc and then LUN 5 would go to path 1 again. Then of course this needed to be set on 20 ESXi servers in the cluster and if any new LUNs were added this would have to be set as well.

Nowadays I would maybe look at doing this with Powershell but back then I was still quite new to it and as I was more comfortable in Unix I used the Vima appliance and setup a simple shell script and the vicfg-mpath command to do the work for me.

Here is the script (click on view source for a better view):

#!/bin/bash
#
# Script to set preferred paths in rotational method
# Set VI_USERNAME and VI_PASSWORD environment variables before running.
#
esxhosts="esxhost1 esxhost2 esxhost3 esxhost4 esxhost5 esxhost6 esxhost7 esxhost8"
echo "===================== Start `date` =========================="
for esxhost in $esxhosts
do
   LUNs=`vicfg-mpath --server $esxhost -b | grep "vmfs/devices/disks" | cut -d\  -f1 | sort -u`
      if [ -z "$LUNs"  ] ; then
         echo "Error getting luns/paths for $esxhost"
      else
         echo "Setting preferred paths for $esxhost "
         for LUN in $LUNs
         do
             lunid=$( echo $LUN | cut -d: -f3)
             pathToSet=$(( $(($lunid % 4)) + 2 ))
             preferredPath=$(echo $LUN | sed "s/vmhba./vmhba${pathToSet}/")
# #debug#       echo Lun = $LUN  lunid = $lunid  path = $pathToSet preferredPath = $preferredPath
             echo Executing vicfg-mpath --server ${esxhost} --policy fixed --lun ${LUN} --path ${preferredPath} --preferred
             vicfg-mpath --server ${esxhost} --policy fixed --lun ${LUN} --path ${preferredPath} --preferred
         done
      fi
echo "----------------------------------------------------------------------------------------"
done