jump to navigation

Network changes April 29, 2010

Posted by vneil in ESXi, network, scripts, VMware.
add a comment

When our environment was first setup it was difficult to get all the necessary buy-in from different departments, for example, the network team.

network pre reconfigThis meant doing things like VLAN tagging / trunking was not feasible, we had to provide connections to 4 different virtual machine networks so we had a lot of network connections and did not have redundancy on all. The servers had 8 network connections but to allow for ESXi / vMotion networks and the VM networks, the maintenance networks for the VMs did not get redundancy plus we had to drop one production network which wasn’t required straight away.

Later we organised an in-house VMware training (given by the excellent Eric Sloof) and invited a couple of the key guys from the network team who we had been working with. This had the great affect of allowing them to see how VMware handled it’s internal network and gave us all a chance to brainstorm ways of changing the network connections as we had also started getting pressure to provide connections to the mainframe production network we had left out.

network after reconfigThe result was a much better architecture and reorganisation of how the adapters were used by VMware and incorporatation of 802.1Q VLAN tagging, proving the benefit of spreading the word and getting buy-in from other departments you deal with.

Seeing as I had 20 ESXi servers in the cluster to change I created a script to do the definition for me. This wasn’t my own work but cobbled together from scripts from Powershell heroes like Alan Renouf and Luc Dekens.

Here is the script to create everything except the Management Network:

$esxserver= Get-VMHost esxserver4.vmware.in
$vmoip = "172.1.101.4"

# Add vmnic4,vmnci6 to vSwitch0
$vSwitch0 = get-virtualswitch -vmhost $esxserver -name vSwitch0
Set-VirtualSwitch -VirtualSwitch $vSwitch0 -Nic vmnic4,vmnic6 -NumPorts 128

# add serverfarm portgroup to vSwitch0
New-VirtualPortGroup -Name "serverfarm" -VirtualSwitch $vSwitch0

# Configure portgroup policies for vSwitch0
$hostview = $esxserver | Get-View
$ns = Get-View -Id $hostview.ConfigManager.NetworkSystem

# set failover policy for serverfarm portgroup
$pgspec = New-Object VMware.Vim.HostPortGroupSpec
$pgspec.vswitchName = "vSwitch0"
$pgspec.Name = "serverfarm"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy
# create object for nic teaming in port group
$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy
$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy
$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic4","vmnic6")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic1")
# load balancing
$pgspec.policy.NicTeaming.policy = "loadbalance_srcid"
# link failure
$pgspec.policy.NicTeaming.failureCriteria = New-Object vmware.vim.HostNicFailureCriteria
$pgspec.policy.NicTeaming.failureCriteria.checkBeacon = $false
# Failback
$pgspec.policy.NicTeaming.RollingOrder = $false
# notify switches
$pgspec.policy.NicTeaming.notifySwitches = $true
$ns.UpdatePortGroup($pgspec.Name,$pgspec)

# set failover policy for Management Network portgroup
$pgspec = New-Object VMware.Vim.HostPortGroupSpec
$pgspec.vswitchName = "vSwitch0"
$pgspec.Name = "Management Network"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy
# create object for nic teaming in port group  (opposite failover to other port group)
$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy
$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy
$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic1")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic4","vmnic6")
# load balancing
$pgspec.policy.NicTeaming.policy = "failover_explicit"
# link failure
$pgspec.policy.NicTeaming.failureCriteria = New-Object vmware.vim.HostNicFailureCriteria
$pgspec.policy.NicTeaming.failureCriteria.checkBeacon = $false
# Failback
$pgspec.policy.NicTeaming.RollingOrder = $false
# notify switches
$pgspec.policy.NicTeaming.notifySwitches = $true
$ns.UpdatePortGroup($pgspec.Name,$pgspec)

# create vSwitch1 for mainframe
New-VirtualSwitch -VMhost $esxserver -Nic vmnic3,vmnic5 -NumPorts 128 -Name "vSwitch1"
$vSwitch1= Get-VirtualSwitch -vmhost $esxserver -Name "vSwitch1"
New-VirtualPortGroup -Name "mainframe" -VirtualSwitch $vSwitch1

# create vSwitch2 for server mgmt and vmotion
New-VirtualSwitch -VMhost $esxserver -Nic vmnic0,vmnic2 -NumPorts 128 -Name "vSwitch2"
$vSwitch1= Get-VirtualSwitch -vmhost $esxserver -Name "vSwitch2"

# create servermgmt_linux (vlan3001)
New-VirtualPortGroup -Name "servermgmt_linux" -VLanId 3001  -VirtualSwitch $vSwitch2

# create servermgmt_win (vlan3002)
New-VirtualPortGroup -Name "servermgmt_win" -VLanId 3002  -VirtualSwitch $vSwitch2

# create vmotion (vlan3099)
New-VMHostNetworkAdapter -VMHost $esxserver -PortGroup vmotion -VirtualSwitch $vSwitch2 -IP $vmoip -SubnetMask 255.255.255.0 -VMotionEnabled $true -EA Stop

$hostview = $esxserver | Get-View
$ns = Get-View -Id $hostview.ConfigManager.NetworkSystem

# set failover policy for vmotion portgroup
$pgspec = New-Object VMware.Vim.HostPortGroupSpec
$pgspec.vswitchName = "vSwitch2"
$pgspec.Name = "vmotion"
$pgspec.vlanId = "3099"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy
# create object for nic teaming in port group
$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy
$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy
$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic2")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic0")
# load balancing
$pgspec.policy.NicTeaming.policy = "failover_explicit"
# link failure
$pgspec.policy.NicTeaming.failureCriteria = New-Object vmware.vim.HostNicFailureCriteria
$pgspec.policy.NicTeaming.failureCriteria.checkBeacon = $false
# Failback
$pgspec.policy.NicTeaming.RollingOrder = $false
# notify switches
$pgspec.policy.NicTeaming.notifySwitches = $true
$ns.UpdatePortGroup($pgspec.Name,$pgspec)

# set failover policy for servermgmt_linux portgroup
$pgspec = New-Object VMware.Vim.HostPortGroupSpec
$pgspec.vswitchName = "vSwitch2"
$pgspec.Name = "servermgmt_linux"
$pgspec.vlanId = "3001"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy
# create object for nic teaming in port group
$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy
$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy
$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic0")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic2")
# load balancing
$pgspec.policy.NicTeaming.policy = "failover_explicit"
# link failure
$pgspec.policy.NicTeaming.failureCriteria = New-Object vmware.vim.HostNicFailureCriteria
$pgspec.policy.NicTeaming.failureCriteria.checkBeacon = $false
# Failback
$pgspec.policy.NicTeaming.RollingOrder = $false
# notify switches
$pgspec.policy.NicTeaming.notifySwitches = $true
$ns.UpdatePortGroup($pgspec.Name,$pgspec)

# set failover policy for servermgmt_win portgroup
$pgspec = New-Object VMware.Vim.HostPortGroupSpec
$pgspec.vswitchName = "vSwitch2"
$pgspec.Name = "servermgmt_win"
$pgspec.vlanId = "3002"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy
# create object for nic teaming in port group
$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy
$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy
$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic0")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic2")
# load balancing
$pgspec.policy.NicTeaming.policy = "failover_explicit"
# link failure
$pgspec.policy.NicTeaming.failureCriteria = New-Object vmware.vim.HostNicFailureCriteria
$pgspec.policy.NicTeaming.failureCriteria.checkBeacon = $false
# Failback
$pgspec.policy.NicTeaming.RollingOrder = $false
# notify switches
$pgspec.policy.NicTeaming.notifySwitches = $true
$ns.UpdatePortGroup($pgspec.Name,$pgspec)

Setting preferred paths in ESXi February 12, 2010

Posted by vneil in ESXi, scripts, VMware.
1 comment so far

Quite a while ago, when I was setting up our current production environment, I had to think about how to set up the multipathing for the SAN LUNs. We have an FC SAN fabric connected to a couple of HDS storage systems and each of our ESXi host servers has 2 dual port FC HBAs. All LUNs would be visible on all four HBAs so each LUN ends up with 4 paths.

To make this simple when setting up the cluster we went for a standard 500GB size for the LUNs and had about 8 LUNs allocated to start with. We needed to somehow level the usage of these LUNs across the 4 paths.

These servers were running ESXi 3.5 and the round robin path policy was still experimental this left a decision of either Fixed or Most Recently Used (MRU) as the multipath policy to use. After some investigation I chose to use Fixed multipath as with MRU you could possibly end up with all LUNs on one path if several path failures go unnoticed.

To simplify the spreading of LUNs over the multiple paths the LUNs were each allocated a path in order, for example LUN 1 to path 1 , LUN 2 to path 2 ..etc and then LUN 5 would go to path 1 again. Then of course this needed to be set on 20 ESXi servers in the cluster and if any new LUNs were added this would have to be set as well.

Nowadays I would maybe look at doing this with Powershell but back then I was still quite new to it and as I was more comfortable in Unix I used the Vima appliance and setup a simple shell script and the vicfg-mpath command to do the work for me.

Here is the script (click on view source for a better view):

#!/bin/bash
#
# Script to set preferred paths in rotational method
# Set VI_USERNAME and VI_PASSWORD environment variables before running.
#
esxhosts="esxhost1 esxhost2 esxhost3 esxhost4 esxhost5 esxhost6 esxhost7 esxhost8"
echo "===================== Start `date` =========================="
for esxhost in $esxhosts
do
   LUNs=`vicfg-mpath --server $esxhost -b | grep "vmfs/devices/disks" | cut -d\  -f1 | sort -u`
      if [ -z "$LUNs"  ] ; then
         echo "Error getting luns/paths for $esxhost"
      else
         echo "Setting preferred paths for $esxhost "
         for LUN in $LUNs
         do
             lunid=$( echo $LUN | cut -d: -f3)
             pathToSet=$(( $(($lunid % 4)) + 2 ))
             preferredPath=$(echo $LUN | sed "s/vmhba./vmhba${pathToSet}/")
# #debug#       echo Lun = $LUN  lunid = $lunid  path = $pathToSet preferredPath = $preferredPath
             echo Executing vicfg-mpath --server ${esxhost} --policy fixed --lun ${LUN} --path ${preferredPath} --preferred
             vicfg-mpath --server ${esxhost} --policy fixed --lun ${LUN} --path ${preferredPath} --preferred
         done
      fi
echo "----------------------------------------------------------------------------------------"
done