HYPER-V: BEST PRACTICES

Tomica Kaniki
tomica@kaniski.eu | http://blog.kaniski.eu/
OUR SPONSORS MADE THIS POSSIBLE!
THANK YOU!
TAKE A DEEP BREATH

"Design/planning phase" is critical!

 HOW MANY
hosts?
CPUs?
RAM?
bandwidth?
?

Microsoft Assessment and Planning Toolkit (MAP Toolkit)

https://www.microsoft.com/en-
us/download/details.aspx?id=7826
DEMO
 HOST (HARDWARE)
use standardized hardware (easier to manage)
use Windows Server certified hardware (https://www.windowsservercatalog.com/)
CPU
server virtualization (Intel VT/AMD-V) and Data Execution Prevention (XD/NX) - Enabled
SLAT-capable with large cache
dont mix vendors (mixed CPU vendors = NO Live Migration)
Hyper-Threading enabled
compatible
performance impact if not using same instruction set
best buy (12-cores per socket?)
RAM
ECC-capable
the more, the better (how many VMs can go down in case of disaster? 30? )
HDD (local)
RAID-1 (of two SSDs or HDDs)
 POWER OPTIONS
full power or green IT?

power settings:
BIOS/UEFI level
host OS level
guest OS level

easy to set:
POWERCFG.EXE /S SCHEME_MIN (High performance)
POWERCFG.EXE /S SCHEME_BALANCED (Balanced (recommended))
 HOST (SOFTWARE) (1)
Hyper-V host is a Hyper-V host (and nothing else)!
install the latest drivers & firmware versions (that work)
use driver/firmware bundles (like SPP for HP servers) to standardize BIOS, NIC,
storage controllers and HBA versions in your environment
http://h17007.www1.hpe.com/us/en/enterprise/servers/products/service_pack/spp/index.aspx

install only the necessary (Hyper-V role, Failover Clustering, MPIO + DSM)
use MinShell or Core (or even Nano ) installation options
or full GUI, configure and then downgrade to Core
or just full GUI
updates & hotfixes
Cluster https://support.microsoft.com/en-us/kb/2920151
Hyper-V https://support.microsoft.com/en-us/kb/3135020
support will first ask if youve installed all the latest and greatest
 HOST (SOFTWARE) (2)
install backup and monitoring agents
install antivirus if necessary (company policy)
dont forget to set exclusions!
dont forget to modify Hyper-V settings after the installation
default paths are on the system drive!
Set-VMHost -ComputerName localhost -VirtualHardDiskPath 'D:\VMs'
Set-VMHost -ComputerName localhost -VirtualMachinePath 'D:\VMs'

sign out from Hyper-V hosts when finished working

we are admins we love PowerShell we dont need RDP
sessions to hosts
remember to close remote PS sessions as well
 ANTIVIRUS EXCLUSIONS
antivirus exclusions for Hyper-V hosts & clusters are:
all directories that contain virtual machines, their disks, snapshots,
processes:
%systemroot%\System32\vmms.exe
%systemroot%\System32\vmwp.exe
%systemroot%\Cluster\clussvc.exe
%systemroot%\Cluster\rhs.exe
folders:
%programfiles%\Microsoft\Hyper-V\*
%public%\Documents\Hyper-V\*
%systemdrive%\ClusterStorage\*
%systemdrive%\ProgramData\Microsoft\Windows\Hyper-V\*
%systemroot%\Cluster\*
(witness disk)
https://support.microsoft.com/en-us/kb/3105657 & https://support.microsoft.com/en-us/kb/961804
 INSTALLATION
manual or automatic installation
speed-up the deployment
unattend.xml (one-time prepare, many-time use )
DEMO
bare-metal host deployment (SCVMM)
MDT/SCCM deployments
which version?
Standard, Datacenter (AVMA)
which option?
full GUI, MinShell, Core, Nano?
full GUI then downgrade to Core?
Remove-WindowsFeature Server-Gui-Shell, Server-Gui-Mgmt-Infra
 MANAGEMENT TOOLS
although real (wo)men use PowerShell for everything, there are
some other tools that can help you: ()
Server Manager console
Best Practices Analyzer (BPA) useful tool!
Hyper-V console
dont use this one for managing clustered VMs please
Failover Cluster console
Cluster Validation Wizard VERY useful tool! (ask Microsoft
Support! )
System Center VMM
it works just fine but not every time ()
3rd party solutions (5nine, SysInternals, )
 VIRTUAL MACHINES (1)
use/convert to Generation 2 VMs where possible
https://blogs.technet.microsoft.com/jhoward/2013/11/1
4/hyper-v-generation-2-virtual-machines-part-10/
Convert-VMGeneration tool
https://code.msdn.microsoft.com/ConvertVMGeneration
use only supported guest OSes/versions
https://technet.microsoft.com/en-
us/library/dn792027(v=ws.11).aspx
update Integration Components regularly
some OS hotfixes bring new versions of ICs as well!
 VIRTUAL MACHINES (2)
dont use screen savers inside VMs and sign out of VMs
templates use the smarter sysprep:
%systemroot%\Sysprep\Sysprep.exe /OOBE /Generalize /Shutdown /Mode:VM
faster sysprep, a lot less hardware recognizing
dont use this templates for multiple virtualization platforms
Convert-WindowsImage
https://gallery.technet.microsoft.com/scriptcenter/Conv
ert-WindowsImageps1-0fe23a8f
offline patching Apply-WindowsUpdate
https://gallery.technet.microsoft.com/Offline-Servicing-
of-VHDs-df776bda
 VIRTUAL MACHINES (3)
devices:
dont use Legacy Network Adapter if its not absolutely necessary
the same goes for virtual HBAs
remove devices youre not using
Automatic Stop Action
do not leave Save change to either Shut Down or Turn Off
disable unnecessary background things:
SuperFetch
Windows Search
Scheduled Tasks (disk defragmentation, )
Aero (optimize for performance)
 ACTIVE DIRECTORY/GPO
should I join Hyper-V hosts to Active Directory domain?
YES (existing or the separate management domain little added
security, but huge management overhead; in highly-secured
environments)
Hyper-V will work if domain is not available (but Live Migration wont!)
should I disable Windows Firewall?
NO (fine-tune it rather!)
Group Policy is a great tool, and your host is domain-joined already, so
GPO disable RDP Printer Mapping
Computer Confguration | Policies | Administrative Templates | Windows
Components | Remote Desktop Services | Remote Desktop Session Host |
Printer Redirection | Do not allow client printer redirection | Enable
 VIRTUAL MEMORY/PAGING FILE
(PAGEFILE.SYS)
size leave it on (automatic) at host level
move it to separate disk
for VM SCSI disk
exclude this disk from Hyper-V Replica
you can use the following:
$computer = Get-WmiObject Win32_ComputerSystem -EnableAllPrivileges
$computer.AutomaticManagedPagefile = $false
$computer.Put()
$CurrentPageFile = Get-WmiObject -Query "select * from Win32_PageFileSetting where
name='C:\\pagefile.sys'"
$CurrentPageFile.delete()
Set-WMIInstance -Class Win32_PageFileSetting -Arguments @{name="D:\pagefile.sys";InitialSize = 0;
MaximumSize = 0}
 CLUSTER
Hyper-V cluster is a Hyper-V cluster (and nothing else)!
start with at least 3 nodes
separate network communication
Corporate, Storage, LiveMigration, vSwitch, Backup
use teaming where it seems appropriate
rename cluster resources (networks, disks) and set network
roles & priority in cluster
Cluster Aware Updating
use it if it makes sense and dont use it with SCCM
http://www.altaro.com/hyper-v/cluster-aware-updating-
hyper-v-basics/
 NETWORKING (1)
(re)name your NICs (and NIC teams and vSwitches)
Local Area Connection 23 is a great name, but
use teaming (its free and it works!)
should you use Active/Actives?
it depends (on underlying network switches and connections
among them possible link saturation)
using 1 NIC in Stand-by is just dumb or not?
Dynamic and Switch Independent is the way to go!
if you have issues with a load balancer, go with Hyper-V port
which architecture should you use? (converged or not?)
http://www.hyper-v.nu/archives/hvredevoort/2014/02/definitive-
guide-to-hyper-v-r2-network-architectures/
 NETWORKING (2)
no need for a dedicated heartbeat network in cluster
all cluster networks communicate heartbeat signals
dont disable VMQ if you dont have problems with it
(Broadcom?)
http://www.dell.com/support/article/us/en/04/SLN132131
create virtual switches
and name them the same (use the convention that works
for you, but same on all hosts)!
how many vSwitches do I need?
smallest number that makes sense (use VLANs!)
(how many NICs? teamed or not? )
 NETWORKING (3)
networking optimizations:
set the right networking order Corporate network first
(ncpa.cpl Advanced Advanced Settings Adapters and Bindings)
disable unnecessary network services
Live Migration leave only Client for Microsoft Networks, File and
Printer Sharing, TCP/IPv4 and TCP/IPv6 (NetBIOS is disabled)
iSCSI leave only TCP/IPv4 and TCP/IPv6 (NetBIOS is disabled)
choose which networks will register in DNS (Corporate only)
set cluster network priorities and roles
choose Live Migration network(s)
iSCSI network use MPIO, not teaming on separate NICs
 NETWORKING (4)
DHCP guard (filtering unsolicited DHCP Server offers)
enable for all VMs, and disable for the ones that need it
small performance impact
Router Guard (filtering ICMP Router Advertisements and
redirect messages)
not used so much
implement IPAM for addressing
instead of those old, boring Excel files
do not share adapter with physical OS
enable Jumbo Frames for CSV, iSCSI and LM networks
 STORAGE (1)
single LUN per VM?
no need CSVs work just fine
SAN or SOFS/SMB3?
if there is SAN in your environment (and its supported for Hyper-V), use it
if you want a cheaper solution (and there is no reason against it), try using
SOFS/SMB3
or a combo (SOFS for SMB3 on SAN LUNs)?
use 4K native disks and 64K allocation unit size for drives hosting VHD(X)
avoid vIDE for data disks (Generation 1 VMs)
use virtual HBAs only if needed
use Storage QoS if you need it
 STORAGE (2)
should I use the witness disk in cluster? Yes.
install MPIO, latest DSMs (usually requires restart) and drivers
rename the CSV mount point (C:\ClusterStorage\Volume1)
put VHD(X)s on CSV into folders!
access-related errors if there are VHD(X)s in a CSV root!
NTFS or ReFS?
NTFS 64k for VHD(X)s for optimal performance
ReFS with Windows Server 2016
measure IOPS
http://blog.workinghardinit.work/2014/01/08/how-to-measure-iops-
of-a-virtual-machine-with-resource-metering-and-measurevm/
 VIRTUAL DISKS
use/convert to VHDX
fixed/dynamic/differencing/pass-through?
VHDX dynamic
VHD fixed-size
dont forget to ensure enough free space for dynamic disks!
the same goes to using thin-provisioned LUNs!
one-liner:
Convert-VHD -Path D:\VM01.vhd -DestinationPath D:\VM01.vhdx -VHDType Dynamic
dont use pass-through disks (no point anymore dynamic disks are
as performant as it gets, and their mobility/manageability is not
questionable
convert pass-through disk to VHDX
New-VHD -Path "D:\VMS\Converted.vhdx" -Dynamic SourceDisk 5
 CLUSTER SHARED VOLUMES (1)
how many and how big?
minimum of 2 (to utilize two different storage controllers, )
< 8 cluster nodes = 1 CSV per 1 node
> 8 cluster nodes = 1 CSV per 2-4 nodes
how many VMs per CSV?
up to 50 (server VMs)
up to 100 (client VMs in VDI environment)
IOPS are all that matters!
enable CSV cache to improve read operations
(Get-Cluster).BlockCacheSize = 2048
metadata changes can only occur on the CSV coordinator (owner node)
please rename CSVs before you use them!
C:\ClusterStorage\Volume1 and C:\ClusterStorage\Volume2 are great
names, but
 CLUSTER SHARED VOLUMES (2)
set the automatic stop action to anything other than Save
fill up your CSVs with a maximum of 75 percent of its capacity to allow
growth of all these files
how much space is available on your CSVs?
http://www.powershellmagazine.com/2014/02/28/reporting-cluster-
shared-volume-csv-disk-space-utilization/
encrypt CSV with BitLocker performance impact of 20-30%
network adapters used for CSVs should have the Client for Microsoft
Networks and File and Printer Sharing for Microsoft Networks enabled
activate Microsoft Failover Cluster Virtual Adapter Performance Filter (not if
you are using Guest Cluster in your virtual machine then it should be
disabled on the host level)
 LIVE MIGRATION (1)
don't forget to set the live migration network and options
use Kerberos constrained delegation instead of CredSSP
(need to log on to the server to start live migration)
enabling KCD for two node cluster is quite easy ():
HYP1 delegates credentials to HYP2
$hvhost = "HYP2"
$domain = "sun.tklabs.eu"
Get-ADComputer HYP1 | Set-ADObject -Add @{"msDS-
AllowedToDelegateTo"="Microsoft Virtual System Migration
Service/$hvhost.$domain", "cifs/$hvhost.$domain","Microsoft Virtual System
Migration Service/$hvhost", "cifs/$hvhost"}
 LIVE MIGRATION (2)
HYP2 delegates credentials to HYP1
$hvhost = "HYP1"
$domain = "sun.tklabs.eu"
Get-ADComputer HYP2 | Set-ADObject -Add @{"msDS-
AllowedToDelegateTo"="Microsoft Virtual System Migration
Service/$hvhost.$domain", "cifs/$hvhost.$domain","Microsoft
Virtual System Migration Service/$hvhost", "cifs/$hvhost"}

enable live migration

Enable-VMMigration -Computername HYP1, HYP2
Set-VMHost -Computername HYP1, HYP2 -
VirtualMachineMigrationAuthenticationType Kerberos

yes, it can be done through GUI as well!

 GUEST CLUSTERING
use Anti-Affinity rules to make sure nodes dont end up on the
same physical host
either VMM or PowerShell
not available through Failover Clustering console!
(Get-ClusterGroup KULEN1).AntiAffinityClassNames = "GuestClusterKULEN"
(Get-ClusterGroup KULEN2).AntiAffinityClassNames = "GuestClusterKULEN"
Get-ClusterGroup KULEN1 | fl Anti*

change default failover-triggering heartbeat times (to allow live migrations on

host, without failover actions in guest default is 10 seconds)
(Get-Cluster).CrossSubnetThreshold = 25
(Get-Cluster).SameSubnetThreshold = 25
 CHECKPOINTS
Windows Server 2012 R2
dont use checkpoints (snapshots) in production
Windows Server 2016
use Production checkpoints
tips:
use them as rarely as possible
checkpoints are no substitution for backups
delete checkpoints as soon as possible
dont delete a checkpoint file on the file level!
console/PowerShell is the only way to go
use them with caution on domain controllers and database servers
 BACKUP AND DISASTER RECOVERY
backup
poor mans backup solution Windows Server Backup
other solutions like SC DPM, Veeam, Altaro,
no real need to backup Hyper-V hosts
except maybe for faster recovery, but
use Hyper-V Replica if you need disaster recovery
its included anyway
exclude Paging file disks from replication
 PERFORMANCE (PERFMON.EXE) (1)
storage
its all about the IOPS!

counters:
\Logical Disk(*)\Avg. Disk sec/Read
\Logical Disk(*)\Avg. Disk sec/Write

thresholds:
up to 15 ms should be OK
15-25 ms may cause negative impact on workloads
> 25 ms will cause negative impact on workloads
 PERFORMANCE (PERFMON.EXE) (2)
memory
there has to be enough memory available inside a VM (or a Hyper-V host)!

counters:
\Memory\Available Mbytes
RAM available for running the active processes
15% (or more) is OK
\Memory\Pages/sec
how often the disk is accessed to resolve hard page faults
< 500 pages per second is OK
> 500 pages per second machine just needs more RAM (dont be cheap!)
 PERFORMANCE (PERFMON.EXE) (3)
network
counters
\Network Interface(*)\Bytes Total/sec
shows current network utilization
20% (or more) free is OK
\Network Interface(*)\Output Queue Length
shows latency in sending network packets (threads that wait on the NIC)
0 is OK
1 or more is a sign of degraded network performance
\Hyper-V Virtual Network Adapter(*)\Bytes/sec
shows which virtual network adapters are consuming the most bandwidth
 PERFORMANCE (PERFMON.EXE) (4)
processor
counters
\Processor(*)\% Processor Time
overall CPU utilization (at host level)
< 80% is always OK ()
\Hyper-V Hypervisor Logical Processor(_Total)\% Total Run Time
at host level, but to evaluate guest utilization
\Hyper-V Hypervisor Root Virtual Processor\% Total Run Time
CPU of the root partition
\Hyper-V Hypervisor Virtual Processor(_Total)\% Total Run Time
total time of the virtual processors
\Hyper-V Hypervisor Virtual Processor(*)\%Guest Run Time
CPU of the guest partitions
always measure the CPU usage of the physical system by using the Hyper-V Hypervisor
Processor performance counters
 PAL IS YOUR NEW (BEST) PAL!
PAL (a.k.a. Performance Analysis of Logs Tool)
reads a performance monitor counter logs and analyzes them using
known thresholds
for Hyper-V, SQL, IIS, Exchange, SharePoint,
you can get it at https://pal.codeplex.com/
free!

(P.S. use en-US regional formatting

(P.P.S. Set-Culture en-US )

DEMO
 A FEW MORE THINGS...
vCPU allocation (Microsoft recommendation)
no more than 8 virtual CPUs per physical CPU core for server workloads
no more than 12 virtual CPUs per physical CPU core for VDI workloads
be careful with thin-provisioned disks (running out of storage space), dynamic memory
(dont use it for SQL or Exchange)
Generation 1 VMs create bigger VHD(X) and smaller partition
you dont need to shutdown VM to resize
dont forget common tools like defrag and chkdsk
set up a naming convention (and stick to it )
cluster log is your friend when things go bad (increase sensitivity logging level)
(Get-Cluster).ClusterLogLevel = 5
Get-ClusterLog -Destination C:\Logs -UseLocalTime
DOCUMENT EVERYTHING.
 RESOURCES
checklists:
https://blogs.technet.microsoft.com/askpfeplat/2013/11/03/windows-server-2012-r2-hyper-
v-best-practices-in-easy-checklist-form/
https://blogs.technet.microsoft.com/askpfeplat/2013/03/10/windows-server-2012-hyper-v-
best-practices-in-easy-checklist-form/
book:
https://www.packtpub.com/virtualization-and-cloud/hyper-v-best-practices
other:
http://www.showit.sk/ShowIT/media/ShowIT/prezentacie/2015/marek_jan_Microsoft-Hyper-
V-performance-tuning.pdf
http://www.altaro.com/hyper-v/23-best-practices-improve-hyper-v-vm-performance/
http://www.altaro.com/hyper-v/19-best-practices-hyper-v-cluster/
http://www.altaro.com/hyper-v/common-hyper-v-deployment-mistakes/
http://www.altaro.com/hyper-v/hyper-v-virtual-cpus-explained/
THANK YOU!