VirtuallyNil

Network changes April 29, 2010

When our environment was first setup it was difficult to get all the necessary buy-in from different departments, for example, the network team.

This meant doing things like VLAN tagging / trunking was not feasible, we had to provide connections to 4 different virtual machine networks so we had a lot of network connections and did not have redundancy on all. The servers had 8 network connections but to allow for ESXi / vMotion networks and the VM networks, the maintenance networks for the VMs did not get redundancy plus we had to drop one production network which wasn’t required straight away.

Later we organised an in-house VMware training (given by the excellent Eric Sloof) and invited a couple of the key guys from the network team who we had been working with. This had the great affect of allowing them to see how VMware handled it’s internal network and gave us all a chance to brainstorm ways of changing the network connections as we had also started getting pressure to provide connections to the mainframe production network we had left out.

The result was a much better architecture and reorganisation of how the adapters were used by VMware and incorporatation of 802.1Q VLAN tagging, proving the benefit of spreading the word and getting buy-in from other departments you deal with.

Seeing as I had 20 ESXi servers in the cluster to change I created a script to do the definition for me. This wasn’t my own work but cobbled together from scripts from Powershell heroes like Alan Renouf and Luc Dekens.

Here is the script to create everything except the Management Network:

$esxserver= Get-VMHost esxserver4.vmware.in
$vmoip = "172.1.101.4"

# Add vmnic4,vmnci6 to vSwitch0
$vSwitch0 = get-virtualswitch -vmhost $esxserver -name vSwitch0
Set-VirtualSwitch -VirtualSwitch $vSwitch0 -Nic vmnic4,vmnic6 -NumPorts 128

# add serverfarm portgroup to vSwitch0
New-VirtualPortGroup -Name "serverfarm" -VirtualSwitch $vSwitch0

# Configure portgroup policies for vSwitch0
$hostview = $esxserver | Get-View
$ns = Get-View -Id $hostview.ConfigManager.NetworkSystem

# set failover policy for serverfarm portgroup
$pgspec = New-Object VMware.Vim.HostPortGroupSpec
$pgspec.vswitchName = "vSwitch0"
$pgspec.Name = "serverfarm"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy
# create object for nic teaming in port group
$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy
$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy
$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic4","vmnic6")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic1")
# load balancing
$pgspec.policy.NicTeaming.policy = "loadbalance_srcid"
# link failure
$pgspec.policy.NicTeaming.failureCriteria = New-Object vmware.vim.HostNicFailureCriteria
$pgspec.policy.NicTeaming.failureCriteria.checkBeacon = $false
# Failback
$pgspec.policy.NicTeaming.RollingOrder = $false
# notify switches
$pgspec.policy.NicTeaming.notifySwitches = $true
$ns.UpdatePortGroup($pgspec.Name,$pgspec)

# set failover policy for Management Network portgroup
$pgspec = New-Object VMware.Vim.HostPortGroupSpec
$pgspec.vswitchName = "vSwitch0"
$pgspec.Name = "Management Network"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy
# create object for nic teaming in port group  (opposite failover to other port group)
$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy
$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy
$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic1")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic4","vmnic6")
# load balancing
$pgspec.policy.NicTeaming.policy = "failover_explicit"
# link failure
$pgspec.policy.NicTeaming.failureCriteria = New-Object vmware.vim.HostNicFailureCriteria
$pgspec.policy.NicTeaming.failureCriteria.checkBeacon = $false
# Failback
$pgspec.policy.NicTeaming.RollingOrder = $false
# notify switches
$pgspec.policy.NicTeaming.notifySwitches = $true
$ns.UpdatePortGroup($pgspec.Name,$pgspec)

# create vSwitch1 for mainframe
New-VirtualSwitch -VMhost $esxserver -Nic vmnic3,vmnic5 -NumPorts 128 -Name "vSwitch1"
$vSwitch1= Get-VirtualSwitch -vmhost $esxserver -Name "vSwitch1"
New-VirtualPortGroup -Name "mainframe" -VirtualSwitch $vSwitch1

# create vSwitch2 for server mgmt and vmotion
New-VirtualSwitch -VMhost $esxserver -Nic vmnic0,vmnic2 -NumPorts 128 -Name "vSwitch2"
$vSwitch1= Get-VirtualSwitch -vmhost $esxserver -Name "vSwitch2"

# create servermgmt_linux (vlan3001)
New-VirtualPortGroup -Name "servermgmt_linux" -VLanId 3001  -VirtualSwitch $vSwitch2

# create servermgmt_win (vlan3002)
New-VirtualPortGroup -Name "servermgmt_win" -VLanId 3002  -VirtualSwitch $vSwitch2

# create vmotion (vlan3099)
New-VMHostNetworkAdapter -VMHost $esxserver -PortGroup vmotion -VirtualSwitch $vSwitch2 -IP $vmoip -SubnetMask 255.255.255.0 -VMotionEnabled $true -EA Stop

$hostview = $esxserver | Get-View
$ns = Get-View -Id $hostview.ConfigManager.NetworkSystem

# set failover policy for vmotion portgroup
$pgspec = New-Object VMware.Vim.HostPortGroupSpec
$pgspec.vswitchName = "vSwitch2"
$pgspec.Name = "vmotion"
$pgspec.vlanId = "3099"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy
# create object for nic teaming in port group
$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy
$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy
$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic2")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic0")
# load balancing
$pgspec.policy.NicTeaming.policy = "failover_explicit"
# link failure
$pgspec.policy.NicTeaming.failureCriteria = New-Object vmware.vim.HostNicFailureCriteria
$pgspec.policy.NicTeaming.failureCriteria.checkBeacon = $false
# Failback
$pgspec.policy.NicTeaming.RollingOrder = $false
# notify switches
$pgspec.policy.NicTeaming.notifySwitches = $true
$ns.UpdatePortGroup($pgspec.Name,$pgspec)

# set failover policy for servermgmt_linux portgroup
$pgspec = New-Object VMware.Vim.HostPortGroupSpec
$pgspec.vswitchName = "vSwitch2"
$pgspec.Name = "servermgmt_linux"
$pgspec.vlanId = "3001"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy
# create object for nic teaming in port group
$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy
$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy
$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic0")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic2")
# load balancing
$pgspec.policy.NicTeaming.policy = "failover_explicit"
# link failure
$pgspec.policy.NicTeaming.failureCriteria = New-Object vmware.vim.HostNicFailureCriteria
$pgspec.policy.NicTeaming.failureCriteria.checkBeacon = $false
# Failback
$pgspec.policy.NicTeaming.RollingOrder = $false
# notify switches
$pgspec.policy.NicTeaming.notifySwitches = $true
$ns.UpdatePortGroup($pgspec.Name,$pgspec)

# set failover policy for servermgmt_win portgroup
$pgspec = New-Object VMware.Vim.HostPortGroupSpec
$pgspec.vswitchName = "vSwitch2"
$pgspec.Name = "servermgmt_win"
$pgspec.vlanId = "3002"
$pgspec.Policy = New-Object VMware.Vim.HostNetworkPolicy
# create object for nic teaming in port group
$pgspec.Policy.NicTeaming = New-Object VMware.Vim.HostNicTeamingPolicy
$pgspec.Policy.NicTeaming.nicOrder = New-Object VMware.Vim.HostNicOrderPolicy
$pgspec.Policy.NicTeaming.nicOrder.activeNic = @("vmnic0")
$pgspec.Policy.NicTeaming.nicOrder.standbyNic = @("vmnic2")
# load balancing
$pgspec.policy.NicTeaming.policy = "failover_explicit"
# link failure
$pgspec.policy.NicTeaming.failureCriteria = New-Object vmware.vim.HostNicFailureCriteria
$pgspec.policy.NicTeaming.failureCriteria.checkBeacon = $false
# Failback
$pgspec.policy.NicTeaming.RollingOrder = $false
# notify switches
$pgspec.policy.NicTeaming.notifySwitches = $true
$ns.UpdatePortGroup($pgspec.Name,$pgspec)