0 calificaciones0% encontró este documento útil (0 votos)
128 vistas79 páginas
VMware vSphere Plan and Design Services Architecture design for Customer Prepared by Jane Q. Consultant, Sr. Consultant VMware Professional Services (c) 2010 VMware, Inc. All rights reserved. Your actual deliverable for your customer will vary depending on the engagement scope, situation, environment, and requirements.
VMware vSphere Plan and Design Services Architecture design for Customer Prepared by Jane Q. Consultant, Sr. Consultant VMware Professional Services (c) 2010 VMware, Inc. All rights reserved. Your actual deliverable for your customer will vary depending on the engagement scope, situation, environment, and requirements.
VMware vSphere Plan and Design Services Architecture design for Customer Prepared by Jane Q. Consultant, Sr. Consultant VMware Professional Services (c) 2010 VMware, Inc. All rights reserved. Your actual deliverable for your customer will vary depending on the engagement scope, situation, environment, and requirements.
Jane Q. Consultant, Sr. Consultant VMware Professional Services consultant@vmware.com VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 2 of 79 Revision History Date Rev Author Comments Reviewers 08 December 2010 v2.2 Pat Carri Pubs Edits/Formatting Jeff Friedman 07 December 2010 v2.1 Andrew Hald Introduced vCAF, reorganized document flow John Arrasjid, Michael Mannarino, Wade Holmes, Kaushik Banerjee 04 November 2010 v2.0 Jeff Friedman vSphere 4.1 Andrew Hald 16 June 2009 v1.0 Mark Ewert, Kingsley Turner, Ken Polakowski Pang Chen
DELETE THE FOLLOWING HIGHLIGHTED TEXT AFTER YOU READ IT This is representative sample deliverable of a Plan and Design for VMware vSphere engagement for use when building, rebuilding, or expanding a specific VMware vSphere design. Your actual deliverable for your customer will vary depending on the engagement scope, situation, environment, and requirements. You will need to update this document based upon your specific customer.
2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. This product is covered by one or more patents listed at http://www.vmware.com/download/patents.html. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
VMware, Inc 3401 Hillview Ave Palo Alto, CA 94304 www.vmware.com
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. VMware is a registered trademark of VMware, Inc. Page 3 of 79 Design Subject Matter Experts The people listed in the following table provided key input into this design. Name Email Address Role/Comments
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 4 of 79 Contents 1. Purpose and Overview ...................................................................... 9 1.1 Executive Summary ....................................................................................................... 9 1.2 Business Background .................................................................................................... 9 1.3 The VMware Consulting and Architecture Framework ................................................ 10 2. Conceptual Design Overview .......................................................... 13 2.1 Customer Requirements.............................................................................................. 13 2.2 Design Assumptions .................................................................................................... 13 2.3 Design Constraints ...................................................................................................... 14 2.4 Use Cases ................................................................................................................... 14 3. vSphere Datacenter Design ............................................................ 15 3.1 vSphere Datacenter Logical Design ............................................................................ 15 3.2 vSphere Clusters ......................................................................................................... 16 3.3 Microsoft Cluster Service in an HA/DRS Environment ................................................ 20 3.4 VMware FT .................................................................................................................. 20 4. VMware ESX/ESXi Host Design ...................................................... 21 4.1 Compute Layer Logical Design ................................................................................... 21 4.2 Host Platform ............................................................................................................... 22 4.3 Host Physical Design Specifications ........................................................................... 23 5. vSphere Network Architecture ......................................................... 24 5.1 Network Layer Logical Design ..................................................................................... 24 5.2 Network vSwitch Design .............................................................................................. 25 5.3 Network Physical Design Specifications ..................................................................... 29 5.4 Network I/O Control ..................................................................................................... 30 6. vSphere Shared Storage Architecture ............................................. 32 6.1 Storage Layer Logical Design ..................................................................................... 32 6.2 Shared Storage Platform ............................................................................................. 33 6.3 Shared Storage Design ............................................................................................... 34 6.4 Shared Storage Physical Design Specifications ......................................................... 36 6.5 Storage I/O Control ...................................................................................................... 36 7. VMware vCenter Server System Design ......................................... 38 7.1 Management Layer Logical Design ............................................................................. 38 7.2 vCenter Server Platform .............................................................................................. 39 VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 5 of 79 7.3 vCenter Server Physical Design Specifications .......................................................... 40 7.4 vCenter Server and Update Manager Databases ....................................................... 40 7.5 Licenses ...................................................................................................................... 43 8. vSphere Infrastructure Security ....................................................... 44 8.1 Overview ...................................................................................................................... 44 8.2 Host Security ............................................................................................................... 44 8.3 vCenter and Virtual Machine Security ......................................................................... 44 8.4 vSphere Port Requirements ........................................................................................ 45 8.5 Lockdown Mode and Troubleshooting Services ......................................................... 45 9. vSphere Infrastructure Monitoring ................................................... 46 9.1 Overview ...................................................................................................................... 46 9.2 Server, Network, and SAN Infrastructure Monitoring .................................................. 46 9.3 vSphere Monitoring ..................................................................................................... 46 9.4 Virtual Machine Monitoring .......................................................................................... 47 10. vSphere Infrastructure Patch/Version Management ...................... 48 10.1 Overview ...................................................................................................................... 48 10.2 vCenter Update Manager ............................................................................................ 48 10.3 vCenter Server and vSphere Client Updates .............................................................. 50 11. Backup/Restore Considerations .................................................... 51 11.1 Hosts............................................................................................................................ 51 11.2 Virtual Machines .......................................................................................................... 51 12. Design Assumptions ...................................................................... 52 12.1 Hardware ..................................................................................................................... 52 12.2 External Dependencies ............................................................................................... 53 13. Reference Documents ................................................................... 54 13.1 Supplemental White Papers and Presentations .......................................................... 54 13.2 Supplemental KB Articles ............................................................................................ 56 13.3 Supplemental KB Articles ............................................................................................ 57 Appendix A ESX/ESXi Host Estimation .............................................. 58 Appendix B ESX/ESXi Host PCI Configuration ................................... 62 Appendix C Hardware BIOS Settings ................................................. 63 Appendix D Network Specifications .................................................... 64 Appendix E Storage Volume Specifications ........................................ 65 VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 6 of 79 Appendix F LUN Sizing Recommendations ........................................ 66 Appendix G Security Configuration ..................................................... 68 Appendix H Port Requirements .......................................................... 70 Appendix I Monitoring Configuration ................................................... 74 Appendix J Naming Conventions ........................................................ 77 Appendix K Design Log ...................................................................... 79
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 7 of 79 List of Figures Figure 1. Datacenter Logical Design ............................................................................................. 17 Figure 2. Network Switch Design .................................................................................................. 27 Figure 3. SAN Diagram ................................................................................................................. 34
List of Tables Table 1. Infrastructure Design Qualities ........................................................................................ 11 Table 2. Infrastructure Design Quality Ratings .............................................................................. 12 Table 3. Customer Requirements.................................................................................................. 13 Table 4. Design Assumptions ........................................................................................................ 13 Table 5. Design Constraints .......................................................................................................... 14 Table 6. Continuous Availability or High Availability ...................................................................... 16 Table 7. Total Number of Hosts and Clusters Required................................................................ 16 Table 8. VMware HA Cluster Configuration .................................................................................. 18 Table 9. Option 1 Name or Option 2 Name ................................................................................... 21 Table 10. VMware ESX/ESXi Specifications ................................................................................. 22 Table 11. VMware ESX/ESXi Host Hardware Physical Design Specifications ............................. 23 Table 12. Option 1 Name or Option 2 Name ................................................................................. 24 Table 13. Proposed Virtual Switches Per Host ............................................................................. 25 Table 14. vDS Configuration Settings ........................................................................................... 28 Table 15. vDS Security Settings .................................................................................................... 29 Table 16. vSwitches by Physical/Virtual NIC, Port and Function .................................................. 29 Table 17. Virtual Switch Port Groups and VLANs ......................................................................... 30 Table 18. Virtual Switch Port Groups and VLANs ......................................................................... 30 Table 19. Option 1 Name or Option 2 Name ................................................................................. 32 Table 20. Shared Storage Logical Design Specifications ............................................................. 33 Table 21. Shared Storage Physical Design Specifications ........................................................... 36 Table 22. Storage I/O Enabled ...................................................................................................... 36 Table 23. Disk Shares and Limits .................................................................................................. 37 Table 24. Option 1 Name or Option 2 Name ................................................................................. 38 Table 25. vCenter Server Logical Design Specifications .............................................................. 39 Table 26. vCenter Server System Hardware Physical Design Specifications .............................. 40 Table 27. vCenter Server and Update Manager Databases Design ............................................. 40 Table 28. vCenter Server and Update Manager Database Names .............................................. 41 VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 8 of 79 Table 29. SQL Server Database Accounts ................................................................................... 42 Table 30. ODBC System DSN ....................................................................................................... 42 Table 31. Lockdown Mode Configurations .................................................................................... 45 Table 32. Estimated Update Manager Storage Requirements ..................................................... 49 Table 33. vCenter Update Manager Specifications ....................................................................... 49 Table 34. Sources of Technical Assumptions for this Design ....................................................... 52 Table 35. VMware Infrastructure External Dependencies ............................................................. 53 Table 36. CPU Resource Requirements ....................................................................................... 58 Table 37. RAM Resource Requirements ....................................................................................... 58 Table 38. Proposed ESX/ESXi Host CPU Logical Design Specifications ..................................... 59 Table 39. Proposed ESX/ESXi Host RAM Logical Design Specifications .................................... 59 Table 40. VMware vSphere Consolidation Ratios ......................................................................... 61 Table 41. ESX/ESXi Host PCIe Slot Assignments ........................................................................ 62 Table 42. ESX/ESXi Hostnames and IP Addresses...................................................................... 64 Table 43. VMFS Volumes .............................................................................................................. 65 Table 44. NFS Volumes ................................................................................................................ 65 Table 45. vSphere Roles and Permissions ................................................................................... 68 Table 46. vCenter Virtual Machine and Template Inventory Folders to be used to Secure VMs . 69 Table 47. ESX/ESXi Port Requirements ....................................................................................... 70 Table 48. vCenter Server Port Requirements ............................................................................... 71 Table 49. vCenter Converter Standalone Port Requirements ....................................................... 72 Table 50. vCenter Update Manager Port Requirements ............................................................... 73 Table 51. SNMP Receiver Configuration ...................................................................................... 74 Table 52. vCenter SMTP Settings ................................................................................................. 74 Table 53. Physical to Virtual Windows Performance Monitor (Perfmon) Counters ....................... 74 Table 54. Modifications to Default Alarm Trigger Types ............................................................... 76 Table 55. Design Log .................................................................................................................... 79
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 9 of 79 1. Purpose and Overview 1.1 Executive Summary This VMware vSphere architecture design was developed to support a virtualization project to consolidate 1,000 existing physical servers. The required infrastructure being defined here will be used not only for the first attempt at virtualization, but also as a foundation for follow-on projects to completely virtualize the enterprise and to prepare it for the journey to Cloud Computing. Virtualization is being adopted to slash power and cooling costs, reduce the need for expensive datacenter expansion, increase operational efficiency, and capitalize on the higher availability and increased flexibility that comes with running virtual workloads. The goal is for IT to be well- positioned to respond rapidly to ever-changing business needs. This document details the recommended vSphere foundation architecture to implement based on a combination of VMware best practices and specific business requirements and goals. The document provides both logical and physical design considerations encompassing all VMware vSphere-related infrastructure components, including requirements and specifications for virtual machines and hosts, networking and storage, and management. After this initial, foundation architecture is successfully implemented, the architecture can be rolled out to other locations to support a virtualization-first initiative, meaning that all future x86 workloads will be provisioned on virtual machines by default. 1.2 Business Background Company and project background: Multinational manufacturing corporation with large retail sales and finance divisions Vision for the future of IT is to use virtualization as a key enabling technology This first foundation infrastructure to be located at a primary U.S. datacenter in Burlington, Massachusetts Initial consolidation project targets 1,000 mission-critical x86 servers
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 10 of 79 1.3 The VMware Consulting and Architecture Framework The VMware Consulting and Architecture Framework (vCAF) is a set of tools for delivering all VMware consulting engagements in a standardized way. The framework guides the design process and creates the architecture design. vCAF is executed in the following phases: Discovery o Understand the customers business requirements and objectives for the project. o Capture the business and technical requirements, assumptions, and constraints for the project. o Perform or secure a current state analysis of customers existing VMware vSphere environment. Development o Conduct design workshops and interviews with the following subject matter experts: application, business continuity (BC) and DR, environment, storage, networking, security, server administration, and operations. The goal of these workshops is to transform the business and technical requirements into a logical design that is scalable and flexible. o Discuss design options with tradeoffs and benefits. Compare the design choices with their impact against key infrastructure qualities as defined by vCAF. Execution o Build the vSphere infrastructure according to the physical design specifications. o Execute verification procedures to confirm operational and technical success. Review o Identify next steps. o Re-calibrate enterprise goals and identify any new or modified objectives. o Match enterprise goals to future engagement of professional services.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 11 of 79 The infrastructure qualities shown in the following table are used to categorize requirements and design decisions as well as assess infrastructure maturity. Table 1. Infrastructure Design Qualities
Design Quality Description Availability Indicates the effect of a design choice on the ability of a technology and the related infrastructure to achieve highly available operation. Key metrics: % uptime Manageability Indicates the effect of a design choice on the flexibility of an environment and the ease of operations in its management. Sub- qualities may include scalability and flexibility. Higher ratios are considered better indicators. Key Metrics: Servers per administrator Clients per IT personnel Time to deploy new technology Performance Indicates the effect of a design choice on the performance of the environment. This does not necessarily reflect the impact on other technologies within the infrastructure. Key Metrics: Response time Throughput Recoverability Indicates the effect of a design choice on the ability to recover from an unexpected incident which affects the availability of an environment. Key Metrics: RTO Recovery Time Objective RPO Recovery Point Objective Security Indicates the effect of a design choice to have a positive or negative impact on overall infrastructure security. Can also indicate whether a quality has an impact on the ability of a business to demonstrate or achieve compliance with certain regulatory policies. Key Metrics: Unauthorized access prevention Data integrity and confidentiality Forensic capabilities in case of a compromise
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 12 of 79 All qualities are rated as shown in the following table. Table 2. Infrastructure Design Quality Ratings Symbol Definition Positive effect on the design quality o No effect on the design quality or there is no comparison basis Negative effect on the design quality
This document captures the design decisions made for the solution to meet customer requirements. In some cases, customer-specific requirements and existing infrastructure limitations or constraints might result in a valid but sub-optimal design choice. The primary goal of this design is to provide a service that corresponds with the business objectives of the organization. With financial constraints factored into the decision process, the key qualities to take into consideration are: Availability Recoverability Performance Manageability Security
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 13 of 79 2. Conceptual Design Overview DELETE THE FOLLOWING HIGHLIGHTED TEXT AFTER YOU READ IT At the top of this section, insert text describing the key motivations for this design effort. What are customer pain points? What are the factors driving business decisions in IT? Fill in the requirements, assumptions and constraints below with information gathered during your engagement. The key customer drivers and requirements guide all design activities throughout an engagement. Requirements, assumptions, and constraints are carefully logged so that all logical and physical design elements can be easily traced back to their source and justification. 2.1 Customer Requirements Requirements are the key demands on the design. Sources include both business and technical representatives. Table 3. Customer Requirements ID Requirement Source Date Approved r101 Tier 1 services must meet a one hour RTO. Fred Jones r102 PCI-compliant services require isolation from other services. David Johnson
2.2 Design Assumptions Assumptions are introduced to reduce design complexity and represent design decisions that are already factored into the environment. Table 4. Design Assumptions ID Assumption Source Date Approved a101 All services belonging to the Billing Department can be considered as Tier 2 for the purposes of DR planning. James Hamilton a102 Performance is considered acceptable if the end user does not notice a difference between the original platform and the new design. David Johnson
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 14 of 79 2.3 Design Constraints Constraints limit the logical design decisions and physical specifications. They are decisions made independent of this engagement that may or may not align with stated objectives. Table 5. Design Constraints ID Constraint Source Date Approved c101 IBM will provide the server hardware. Fred Jones c102 Qlogic HBAs will be used in the ESX hosts. Fred Jones
2.4 Use Cases This design is targeted at the following use cases: Server consolidation (power and cooling savings, green computing) Server infrastructure resource optimization (load balancing, high availability) Rapid provisioning Server standardization The following use cases are deferred to a future project: Server containment (new workloads)
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 15 of 79 3. vSphere Datacenter Design 3.1 vSphere Datacenter Logical Design In VMware vSphere, a datacenter is the highest level logical boundary. The datacenter may be used to delineate separate physical sites/locations or vSphere infrastructures with completely independent purposes. Within vSphere datacenters, VMware ESX/ESXi hosts are typically organized into clusters. Clusters group similar hosts into a logical unit of virtual resources, enabling such technologies as VMware vMotion, High Availability (HA), Dynamic Resource Scheduling (DRS), and VMware Fault Tolerance (FT). To address customer requirements, the following design options were proposed during the design workshops. For each design decision, the impact on each infrastructure quality is noted. The selected design option is then explained with the appropriate justification. DELETE THE FOLLOWING HIGHLIGHTED GUIDANCE TEXT AFTER YOU READ IT AND REMOVE THE HIGHLIGHTING FROM THE DESIGN DECISION TEMPLATE. The following design decision is an example. Follow the model below to communicate the design decisions appropriate to your customer and their requirements. 3.1.1 Tier 2 Service Availability Customer requirements have not explicitly defined the service level availability for Tier 2 services. The following two options are available. 3.1.1.1. Option 1: Continuous Availability Continuously available services: Are redundant at the application level Have no single points of failure The drawbacks of continuous availability are: More expensive infrastructure Not-all applications are compatible with continuous availability methods 3.1.1.2. Option 2: High Availability Highly available services: Limit single points of failure Are less expensive to support The drawbacks of high availability are: Some service downtime is possible Application awareness requires additional effort Tier 2 services are not defined as mission critical, but they are still important to daily business operations of the customer. Providing no service availability efforts is not an acceptable option. Budgetary constraints must be balanced with availability requirements so that if a problem occurs, the service is restored within one hour, as stated in the service level agreement for Tier 2 services (r007).
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 16 of 79 Table 6. Continuous Availability or High Availability Design Quality Option 1 Option 2 Comments Availability Both options improve availability, though Option 1 would guarantee a higher level. Manageability o Option 1 would be harder to maintain due to increased complexity. Performance o o Both design options have no impact on performance Recoverability Both options improve recoverability Security o o Both design options have no impact on security Legend: = positive impact on quality; = negative impact on quality; o = no impact on quality Due to the lower cost and better manageability, customer has selected Option 2, High Availability. This design decision is reflected in the physical design specifications outlined below. 3.2 vSphere Clusters As part of this logical design, vSphere clusters are created to aggregate hosts. The number of hosts per cluster is shown in the following table. Table 7. Total Number of Hosts and Clusters Required Attribute Specification Number of hosts required to support 1,000 VMs 17 Approximate number of VMs per host 58.82 Maximum vSphere cluster size if hosts support more than 40 VMs each 16 hosts Capacity for host failures per cluster 1 host Dedicated hosts for maintenance capacity per cluster 1 host Number of usable hosts per cluster 6 hosts Number of clusters created 3 Total usable capacity in hosts 18 hosts Total usable capacity in VMs (total usable hosts * true consolidation ratio) 1,084
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 17 of 79 Assumptions and Caveats The total usable capacity in VMs does not account for shadow instances of VMs needed to support VMware FT. The number of protected VMs must be counted double. Host failures for VMware HA are expressed in the number of allowable host failures, meaning that the expected load should be able to run on surviving hosts. HA policies can also be applied on a percentage spare capacity basis. Dedicated hosts for maintenance assumes that a host is reserved strictly to offload running VMs from other hosts that must undergo maintenance. When not being used for maintenance, such hosts can also provide additional spare capacity to support a second host failure, or for unusually high demands on resources. Having dedicated maintenance hosts can be considered somewhat conservative, as spare capacity is being allocated strictly for maintenance activities. Such spare capacity is earmarked here to ensure that there is sufficient capacity to run VMs with minimal disruption. Hosts are evenly divided across the clusters. Clusters can be created and organized to enforce resource allocation policies such as: o Load balancing o Power management o Affinity rules o Protected workloads o Limited number of licenses available for specific applications Figure 1. Datacenter Logical Design Datacenter A Datacenter B None Tier One Cluster Host & VM Affinity Rules Fault Tolerance vCenter Server VM (with VUM, Converter plug-ins) Tier Two Cluster Distributed Power Management Tier Three Cluster Application Cluster for limited licenses Hosts 1-8 Hosts 9-16 Hosts 17-24 MS SQL Server DB (vCenter DB, VUM DB) Active Directory
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 18 of 79 3.2.1 VMware HA Each cluster are configured for VMware High Availability (HA) to automatically recover VMs if either ESX/ESXi host fails, or if there is an individual VM failure. A host is declared failed if the other hosts in the cluster cannot communicate with it. A VM is declared failed if a heartbeat inside the guest OS can no longer be received. VMs are tiered in relative order of priority for restarts: High (for example, Windows Active Directory domain controller VMs). Medium (default). Low. Disabled. For non-critical VMs. Do not restart. Sacrifices resources for higher priority VMs (for example, QA and test VMs). The configuration settings for VMware HA are shown in the following table. Table 8. VMware HA Cluster Configuration Attribute Specification Enable host monitoring Enable Admission control Prevent VMs from being powered on if they violate availability constraints Admission control policy Cluster tolerates one host failure Default VM restart priority High (critical VMs) Medium (majority of VMs) Disabled (non-critical VMs) Host isolation response Power off VM Enable VM monitoring Enable VM monitoring sensitivity Medium
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 19 of 79 Setting Explanations Enable host monitoring. When HA is enabled, hosts in the cluster are monitored. If there is a host failure, the virtual machines on a failed host are restarted on alternate running hosts in the cluster. Admission control. Enforces availability constraints and preserves host failover capacity. Any operation on a virtual machine that decreases the unreserved resources in the cluster and violates availability constraints is not permitted. Admission control policy. Each HA cluster can support as many host failures as specified. Default VM restart priority. The priority level specified here is relative. VMs must be assigned a relative restart priority level for HA. VMs are organized into four categories: high, medium, low, and disabled. It is presumed that the majority of systems will be satisfied by the medium setting and are therefore left at default. VMs identified as high priority, such as the Active Directory VMs, are started before the medium priority VMs, which in turn are restarted before the VMs configured with low priority. If insufficient cluster resources are available, it is possible that VMs configured with low priority will not be restarted. To help prevent this situation, non-critical systems, such as QA and test VMs, are set to disabled. If there is a host failure, these VMs are not restarted, saving critical cluster resources for higher priority VMs. Host isolation response. Host isolation response determines what happens when a host in a VMware HA cluster loses its service console/management network connection, but continues running. A host is deemed isolated when it stops receiving heartbeats from all other hosts in the cluster and it is unable to ping its isolation addresses. When this occurs, the host executes its isolation response. To prevent the potential for multiple instances of each virtual machine to be running if a host becomes isolated from the network (causing other hosts to believe it has failed and automatically restart the hosts VMs), the VMs are automatically powered off upon host isolation. Enable VM monitoring. In addition to determining if a host has failed, HA can also monitor for virtual machine failure. When set to enabled, the VM monitoring service (using VMware Tools) evaluates whether each virtual machine in the cluster is running by checking for regular heartbeats from the VMware Tools process running in each guest OS. If no heartbeats are received, HA assumes that the guest operating system has failed, and HA reboots the VM. VM monitoring sensitivity. This affects how relatively quickly HA concludes that a VM failed. Highly sensitive monitoring results in a more rapid conclusion that a failure occurred. While unlikely, highly sensitive monitoring may lead to false identification of failures when the virtual machine in question is actually still working, but heartbeats have not been received due to factors such as resource constraints or network issues. Low sensitivity monitoring allows for more time before HA deems a VM to have failed. At the medium setting, HA restarts the VM if the heartbeat between the host and the VM was not received within a 60 second interval. HA also only restarts the VM after each of the first three failures every 24 hours to prevent repeated failed restarting of VMs that need intervention to recover.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 20 of 79 3.3 Microsoft Cluster Service in an HA/DRS Environment Microsoft Cluster Service (MSCS) applies to Microsoft Cluster Service with Windows Server 2003 and Failover Clustering with Windows Server 2008. All hosts that run MSCS virtual machines are managed by a VMware vCenter Server system with VMware HA and DRS enabled. For a cluster of virtual machines on one physical host, affinity rules are used. For a cluster of virtual machines across physical hosts, anti-affinity rules are used. The advanced option for VMware DRS, ForceAffinePoweron, is set to 1, which enables strict enforcement of the affinity and anti-affinity rules that are created. The automation level of all virtual machines in an MSCS cluster are set to Partially Automated. Note Migration of MSCS clustered virtual machines is not recommended. 3.4 VMware FT Each cluster also supports VMware Fault Tolerance (FT) to protect select critical VMs. The systems to protect are: 2 Blackberry Enterprise servers 2 Microsoft Exchange front-end servers 2 Reporting servers These six systems are distributed evenly amongst the three clusters resulting in two FT-protected virtual machines per eight hosts initially. All VMs to be protected by VMware FT have only one vCPU and disks configured eager-zeroed, also called thick-provisioned (not thin-provisioned). An eager-zeroed thick disk has all space allocated and zeroed out at creation time; this takes a bit longer for the creation time, but facilitates optimal performance and better security. VMware Fault Tolerance with VMware Distributed Resource Scheduler (DRS) are enabled. This process allows fault tolerant virtual machines to benefit from better initial placement and to be included in the cluster's load balancing calculations. Note Enable the Enhanced vMotion Compatibility (EVC) feature. On-Demand Fault Tolerance is scheduled for the two reporting servers during the quarter-end report period and then returned to the HA cluster during non-critical operations. FT traffic is supported with a pair of Gigabit Ethernet ports (see Section 5, vSphere Network Architecture). Because a pair of Gigabit Ethernet ports can support on average 4 to 5 FT- protected VMs per host, there is capacity for additional VMs to be protected by VMware FT.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 21 of 79 4. VMware ESX/ESXi Host Design 4.1 Compute Layer Logical Design The compute layer of the architecture design encompasses the CPU, memory, and hypervisor technology components. Logical design at this level centers on performance and security. To address customer requirements, the following design options were proposed during the design workshops. For each design decision, the impact on each infrastructure quality is noted. The selected design option is then explained with the appropriate justification. DELETE THE FOLLOWING HIGHLIGHTED GUIDANCE TEXT AFTER YOU READ IT AND REMOVE THE HIGHLIGHTING FROM THE DESIGN DECISION TEMPLATE. The following Design Decision is an example. Please follow the model below to communicate the design decisions appropriate to your customer and their requirements. See Section 3.1 for an example. 4.1.1 Design Decision 1 Description of the design decision 4.1.1.1. Option 1: Name Advantages: Advantage 1 Advantage 2 Drawbacks: Drawback 1 Drawback 2 4.1.1.2. Option 2: Name Advantages: Advantage 1 Advantage 2 Drawbacks: Drawback 1 Drawback 2 Further details should be included here. Also highlight any relevant requirements, assumptions and/or constraints that will impact this decision. Table 9. Option 1 Name or Option 2 Name Design Quality Option 1 Option 2 Comments Availability Both options improve availability, though Option 1 would guarantee a higher level. Manageability o Option 1 would be harder to maintain due to increased complexity. VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 22 of 79 Performance o o Both design options have no impact on performance Recoverability Both options improve recoverability Security o o Both design options have no impact on security Legend: = positive impact on quality; = negative impact on quality; o = no impact on quality Which option was selected and why? 4.2 Host Platform This section details the VMware ESX/ESXi hosts proposed for the vSphere infrastructure design. The logical components specified are required by the vSphere architecture to meet calculated consolidation ratios, protect against failure through component redundancy, and support all necessary vSphere features. Table 10. VMware ESX/ESXi Specifications Attribute Specification Host type and version ESXi 4.1 Installable Number of CPUs Number of cores Total number of cores Processor speed 4 4 16 2.4 GHz (2400 MHz) Memory 32GB Number of NIC ports 10 Number of HBA ports 4
VMware ESXi was selected over VMware ESX because of its smaller running footprint, reduced management complexity, and significantly fewer number of anticipated software patches. The exact ESXi installable build version to be deployed will be selected closer to implementation and will be chosen based on the available stable and supported released versions at that time.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 23 of 79 4.3 Host Physical Design Specifications This section details the physical design specifications of the host and attachments corresponding to the previous section that describes the logical design specifications. Table 11. VMware ESX/ESXi Host Hardware Physical Design Specifications Attribute Specification Vendor and model x64 vendor and model Processor type Total number of cores Quad core x64-vendor CPU 16 Onboard NIC vendor and model Onboard NIC ports x speed Number of attached NICs NIC vendor and model Number of ports/NIC x speed Total number of NIC ports NIC vendor and model 2 x Gigabit Ethernet 4 (excluding onboard) NIC vendor and model 2 x Gigabit Ethernet 10 Storage HBA vendor and model Storage HBA type Number of HBAs Number of ports/HBA x speed Total number of HBA ports HBA vendor and model Fibre Channel 2/4GB 2 x 4GB 4 Number and type of local drives RAID level Total storage 2 x Serial Attached SCSI (SAS) RAID 1 (Mirror) 72GB System monitoring IPMI-based BMC
The configuration and assembly process for each system is standardized, with all components installed the same on each host. Standardizing not only the model, but also the physical configuration of the ESX/ESXi hosts, is critical to providing a manageable and supportable infrastructureit eliminates variability. Consistent PCI card slot location, especially for network controllers, is essential for accurate alignment of physical to virtual I/O resources. Appendix B contains further information on the host PCI placement. All ESX/ESXi host hardware including CPUs was selected following the vSphere Hardware Compatibility Lists and the CPUs were determined to be compatible with Fault Tolerance.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 24 of 79 5. vSphere Network Architecture 5.1 Network Layer Logical Design The network layer encompasses all network communications between virtual machines, vSphere management layer, and the physical network. Key infrastructure qualities often associated with networking include availability, security, and performance. To address customer requirements, the following design options were proposed during the design workshops. For each design decision, the impact on each infrastructure quality is noted. The selected design option is then explained with the appropriate justification. DELETE THE FOLLOWING HIGHLIGHTED GUIDANCE TEXT AFTER YOU READ IT AND REMOVE THE HIGHLIGHTING FROM THE DESIGN DECISION TEMPLATE. The following Design Decision is an example. Please follow the model below to communicate the design decisions appropriate to your customer and their requirements. See Section 3.1 for an example. 5.1.1 Design Decision 1 Description of the design decision 5.1.1.1. Option 1: Name Advantages: Advantage 1 Advantage 2 Drawbacks: Drawback 1 Drawback 2 5.1.1.2. Option 2: Name Advantages: Advantage 1 Advantage 2 Drawbacks: Drawback 1 Drawback 2 Further details should be included here. Also highlight any relevant requirements, assumptions and/or constraints that will impact this decision. Table 12. Option 1 Name or Option 2 Name Design Quality Option 1 Option 2 Comments Availability Both options improve availability, though Option 1 would guarantee a higher level. VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 25 of 79 Manageability o Option 1 would be harder to maintain due to increased complexity. Performance o o Both design options have no impact on performance Recoverability Both options improve recoverability Security o o Both design options have no impact on security Legend: = positive impact on quality; = negative impact on quality; o = no impact on quality Which option was selected and why? 5.2 Network vSwitch Design Following best practices, the network architecture complies with these design decisions: Separate networks for vSphere management, VM connectivity, vMotion traffic, and Fault Tolerance logging (VM record/replay) traffic Separate network for NFS or iSCSI (storage over IP) used to store VM templates and guest OS installation ISO files Network I/O control implemented to allow flexible partitioning of network traffic for virtual machine, vMotion, FT, and IP storage traffic across the physical NIC bandwidth Redundant virtual Distributed Switches (vDS) with at least three active physical adapter ports Redundancy across different physical adapters to protect against NIC or PCI slot failure Redundancy at the physical switch level Table 13. Proposed Virtual Switches Per Host Virtual Standard (vSS) or Virtual Distributed Switch (vDS) Function Number of Physical NIC Ports vSS0 Management Console and vMotion 2 vDS1 VM, Storage over IP and FT 6
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 26 of 79 vSS 0 is dedicated to the management network and vMotion. Service consoles and VMkernel ports (used for vMotion and IP storage) do not migrate from host to host; these can remain on a Virtual Standard Switch (vSS.) vDS1, is allocated to virtual machine, IP Storage, and Fault Tolerance network traffic. This should be configured to a Disturbed Switch to take advantage of network I/O, Load Based Teaming, and Network vMotion. This vDS is configured to use six active Ethernet adapters. All physical network switch ports connected to these adapters are configured as trunk ports with spanning tree disabled. The trunk ports are configured to pass traffic for all VLANs used by the virtual switch. The physical NIC ports are connected to redundant physical switches. No traffic shaping policies are in place, Load-based teaming is configured for improved network traffic distribution between the pNICs and Network I/O Control enabled. VM network connectivity uses virtual switch port groups and 802.1q VLAN tagging to segment traffic into four VLANs To support the network demands of up to 60 VMs per host, this vDS is configured to use six active Gigabit Ethernet adapters. All physical network switch ports connected to these adapters are configured as trunk ports with spanning tree disabled. The trunk ports are configured to pass traffic for all VLANs used by the virtual switch. The physical NIC ports are connected to redundant physical switches.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 27 of 79 Figure 2. Network Switch Design ESX/ESXi Host Virtual Standard Switch0 Virtual Distributed Switch1 vmnic7 Onboard vmnic5 Slot 7 vmnic6 Onboard vmnic4 Slot 7 vmnic2 Slot 7 Standby vmnic0 Onboard Active vmnic1 Onboard vmnic3 Slot 7 Physical Switch1 VMkernel: vMotion Port Group Management Console Port Group Physical Switch2 Virtual Machine Network Port Group VMkernel: Storage over IP Port Group VMkernel: Fault Tolerance Port Group VLAN 100 VLAN 500 VLAN 200 VLAN 300 VLAN 600 VLAN 650
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 28 of 79 Table 14. vDS Configuration Settings Parameter Setting Load balancing Route based on physical NIC load Failover detection Beacon probing Notify switches Enabled Rolling failover No Failover order All active
vSwitch Configuration Setting Explanations Load Balancing. Route based on physical NIC ensures vDS dvUplink capacity is optimized. Load-Based Teaming (LBT) avoids the situation of other teaming policies in which some of the dvUplinks in a DV Port Groups team were idle while others were completely saturated just because the teaming policy used is statically determined. LBT reshuffles port binding dynamically based on load and dvUplinks usage to make an efficient use of the available bandwidth. LBT only moves ports to dvUplinks configured for the corresponding DV Port Groups team. Note that LBT does not use shares or limits to make its determination while rebinding ports from one dvUplink to another. Failover Detection. In addition to link status, the VMkernel sends out and listens for periodic beacon probes on all network adapters in the team. This enhances link status, which relies exclusively on link integrity of the physical network adapter to determine when a failure occurs. Link status enhanced by beacon probing detects failures that are due to cable disconnects or physical switch power failures, as well as configuration errors or network interruptions beyond the local NIC termination point. Notify Switches. When enabled, this option sends out a gratuitous ARP whenever a new NIC is added to the team or when a virtual NIC begins using a different physical uplink on the ESX/ESXi host. This option helps to lower latency issues when a failover occurs or when virtual machines are migrated to another host using vMotion. Rolling Failover. Determines how a physical adapter is returned to active duty after recovering from a failure. When set to No, the adapter is returned to active duty immediately upon recovery. Setting it to Yes keeps the adapter inactive, even if it is recovered and requires manual intervention to return it to service. Failover Order. All physical adapters assigned to each vSwitch and port group are configured as Active adapters. No adapters are configured as standby or unused.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 29 of 79 Table 15. vDS Security Settings Parameter Setting Promiscuous mode Reject (default) MAC address changes Reject Forged transmits Reject
vDS Security Setting Explanations Promiscuous Mode. Setting to Reject at the vSwitch level protects against virtual machine virtual network adapters. Placing a VM virtual network adapter in promiscuous mode has no effect on which frames are received by the adapter. MAC Address Changes. Setting to Reject at the vSwitch level protects against MAC address spoofing. If the guest OS changes the MAC address of the adapter to anything other than what is in the .vmx configuration file, all inbound frames are dropped if the guest OS changes the MAC address back to match the MAC address in the .vmx configuration file, inbound frames are sent again. Forged Transmits. Setting to Reject at the vSwitch level protects against MAC address spoofing. Outbound frames with a source MAC address that is different from the one set on the adapter are dropped. 5.3 Network Physical Design Specifications This section expands on the logical network design in the corresponding previous section by providing details on the physical NIC layout and physical network attributes. Table 16. vSwitches by Physical/Virtual NIC, Port and Function vSwitch vmnic NIC / Slot Port Function 0 0 Onboard
N/A 0 Management Console and vMotion 0 2 1 Management Console and vMotion 1 1 Onboard
Quad PCIe Slot 7 0 VM, FT, and Storage over IP traffic 1 3 1 VM, FT, and Storage over IP traffic 1 5 Quad PCIe Slot 7
Onboard
0 VM, FT, and Storage over IP traffic 1 7 1 VM, FT, and Storage over IP traffic 1 4 Quad PCIe Slot 7
Onboard 0 VM, FT, and Storage over IP traffic 1 6 1 VM, FT, and Storage over IP traffic
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 30 of 79 Table 17. Virtual Switch Port Groups and VLANs vSwitch Port Group Name VLAN ID 0 MGMT-100 100 0 VMOTION-500 500 1 PROD-200 200 0 DEV-300 300 0 FT-600 600 0 NFS-650 650 0 N/A 1
See Appendix D for more information on the physical network design specifications. 5.4 Network I/O Control Virtual Distributed Swtich1 (vDS1) are configured with Network I/O Control enabled. After Network I/O control is enabled, traffic through that virtual distributed switch is divided into the following network resource pools: FT traffic, iSCSI traffic, vMotion traffic, management traffic, NFS traffic and virtual machine traffic. This design specifies that virtual machine, iSCSI, and FT network traffic are dedicated to virtual distributed swtich1. The priority of the traffic from each of these network resource pools is set by the physical adapter shares and host limits for each network resource pool. Virtual machine traffic is set to High, the FT resource pool set to Normal, and the iSCSI traffic set to Low. These reservations apply only when the physical adapter is saturated. Note The iSCSI traffic resource pool shares do not apply to iSCSI traffic on a dependent hardware iSCSI adapter. Table 18. Virtual Switch Port Groups and VLANs Network Resource Pool Physical Adapter Shares Host Limit Fault Tolerance Normal Unlimited iSCSI Low Unlimited Management N/A N/A NFS N/A N/A Virtual Machine High Unlimited vMotion N/A N/A
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 31 of 79 Network I/O Settings Explanation Host Limits. Host limits are the upper limit of bandwidth that the network resource pool can use. Physical Adapter Shares. Shares assigned to a network resource pool determine the total available bandwidth guaranteed to the traffic associated with that network resource pool. o High. Sets the shares for this resource pool to 100. o Normal. Sets the shares for this resource pool to 50. o Low. Sets the shares for this resource pool to 25. o Custom. A specific number of shares, from 1 to 100, for this network resource pool.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 32 of 79 6. vSphere Shared Storage Architecture 6.1 Storage Layer Logical Design To address customer requirements, the following design options were proposed during the design workshops. For each design decision, the impact on each infrastructure quality is noted. The selected design option is then explained with the appropriate justification. DELETE THE FOLLOWING HIGHLIGHTED GUIDANCE TEXT AFTER YOU READ IT AND REMOVE THE HIGHLIGHTING FROM THE DESIGN DECISION TEMPLATE. The following Design Decision is an example. Please follow the model below to communicate the design decisions appropriate to your customer and their requirements. See Section 3.1 for an example. 6.1.1 Design Decision 1 Description of the design decision 6.1.1.1. Option 1: Name Advantages: Advantage 1 Advantage 2 Drawbacks: Drawback 1 Drawback 2 6.1.1.2. Option 2: Name Advantages: Advantage 1 Advantage 2 Drawbacks: Drawback 1 Drawback 2 Further details should be included here. Also highlight any relevant requirements, assumptions and/or constraints that will impact this decision. Table 19. Option 1 Name or Option 2 Name Design Quality Option 1 Option 2 Comments Availability Both options improve availability, though Option 1 would guarantee a higher level. Manageability o Option 1 would be harder to maintain due to increased complexity. VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 33 of 79 Performance o o Both design options have no impact on performance Recoverability Both options improve recoverability Security o o Both design options have no impact on security Legend: = positive impact on quality; = negative impact on quality; o = no impact on quality Which option was selected and why? 6.2 Shared Storage Platform This section details the shared storage proposed for the vSphere infrastructure design. Table 20. Shared Storage Logical Design Specifications Attribute Specification Storage type Fibre Channel SAN Number of storage processors 2 (redundant) Number of switches Number of ports per host per switch 2 (redundant) 2 LUN size 1TB Total LUNs 50 VMFS datastores per LUN 1 VMFS version 3.33
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 34 of 79 6.3 Shared Storage Design The following figure illustrates the design. Figure 3. SAN Diagram ESX/ESXi Host vmhba0 vmhba1 Fibre Switch A Fibre Switch B SAN Storage Processor A SAN Storage Processor B 1TB LUN VMFS 1 VMFS 2 VMFS 3 VMFS ... VMFS ... VMFS ... VMFS ... VMFS 50 1TB LUN 1TB LUN 1TB LUN 1TB LUN 1TB LUN 1TB LUN 1TB LUN vmhba2 vmhba3
Based on the results of the physical system assessment, it was determined that on average, each system has a 36GB system volume with 9 gigabytes used, and a 143GB data volume with 22 gigabytes used. After projecting for volume growth and providing 33% minimum free space per volume, for the task of estimating overall storage requirements it was determined that each virtual machine would be configured with a 12GB system volume and a 40GB data volume. Unless constrained by specific application or workload requirements, or special circumstances (such as being protected by VMware Fault Tolerance), all data volumes are provisioned as thin disks with the system volumes deployed as thick. This strategic over-provisioning saves an estimated 8.25TB of storage, assuming that on average, 50% of the currently available storage on each data volume remains unused. The consumption of each storage volume is monitored in production with alarms configured to alert if any approach capacity to provide sufficient time to source and provision additional disk.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 35 of 79 With the intent to maintain at least 15% free capacity on each VMFS volume for VM swap files, snapshots, logs, and thin volume growth, it was determined that 48.7TB of available storage is required to support 1,000 virtual machines after accounting for the long term benefit of thin provisioning. This will be provided in 50 1TB LUNs (the additional 1.23 TB is for growth and test capacity). These LUNs will be zoned to all hosts and formatted as 50 VMFS datastores. 1TB was selected because it provides the best balance between performance and manageability with approximately 20 VMs and 40 virtual disks per volume. Although larger LUNs up to 2TB (without extents) and 64TB (with extents) are possible, this size was chosen for several reasons. For manageability, it allows an adequately large portion of disks to better use resources and limit storage sprawl. A smaller size maintains a reasonable RTO and reduces the risks associated with losing a single LUN. In addition, the size limits the number of VMs that remain on a single LUN. Additionally, three NFS volumes are presented to each ESX/ESXi host for the storage of virtual machine templates, guest operating system installation, CD images (ISOs), and to provide administrators second-tier storage for log and VM archival and infrastructure testing. The separation of such files from VM files was done recognizing that these non-VM files can often have higher I/O characteristics. Each ESX/ESXi host is to be provisioned with a vSS switch supported by two physical Gigabit Ethernet adapters dedicated for IP storage connectivity.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 36 of 79 6.4 Shared Storage Physical Design Specifications This section details the physical design specifications of the shared storage corresponding to the previous section that describes the logical design specifications. Table 21. Shared Storage Physical Design Specifications Attribute Specification Vendor and model Storage vendor and model. Type Active/passive. ESX/ESXi host multipathing policy Most Recently Used (MRU). Set because the SAN is an active/passive array to avoid path thrashing. With MRU, a single path to the SAN is used until it becomes inactive, at which point it switches to another path and continues to use this new path until it fails; the preferred path setting is disregarded. Minimum/Maximum speed rating of switch ports 2GB/4GB.
See Appendix E for an inventory of VMFS and NFS volumes. 6.5 Storage I/O Control Storage I/O Control is enabled. This allows cluster-wide storage I/O prioritization, providing the ability to control the amount of storage I/O that is allocated to virtual machines during periods of I/O congestion. The shares are set per virtual machine and can be adjusted for each VM based on need. Table 22. Storage I/O Enabled Datastore Path Storage I/O Enabled Prod_san01_02 vmhba1:0:0:3 /dev/sda3 48f85575-5ec4c587- b856-001a6465c102 yes Prod_san01_07 vmhba2:0:4:1 /dev/sdc1 48fbd8e5-c04f6d90- 1edb-001cc46b7a18 yes Prod_san01_37 vmhba32:0:1:1 /dev/sde1 48fe2807-7172dad8- f88b-0013725ddc92 yes Prod_san01_44 vmhba32:0:0:1 /dev/sdd1 48fe2a3d-52c8d458- e60e-001cc46b7a18 yes
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 37 of 79 Table 23. Disk Shares and Limits VM Disk Shares Limit IOPS ds007 Disk 1 Low 500 kf002 Disk 1 and 2 Normal 1000 jf001 Disk 1 High 2000 rs003 Disk 1 and 2 Custom 350
Shared I/O Control Settings Explanation Storage I/O Enabled. Storage I/O is enabled per datastore. Navigate to the Configuration Tab > Properties to verify that the feature was enabled. Storage I/O Shares. Storage I/O shares are similar to VMware CPU and memory shares. Shares define the hierarchy of the virtual machines for distribution of storage I/O resources during periods of I/O congestion. Virtual machines with higher shares have higher throughput and lower latency. Limit IOPs. By default IOPS allowed for a virtual machine are unlimited. By allocating storage I/O resources, you then limit the IOPS allowed to a virtual machine. If a machine has multiple disks, you must set the same IOPS value for all the disks that access that virtual machine.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 38 of 79 7. VMware vCenter Server System Design 7.1 Management Layer Logical Design The Management Layer To address customer requirements, the following design options were proposed during the design workshops. For each design decision the impact on each infrastructure quality is noted. The selected design option is then explained with the appropriate justification. DELETE THE FOLLOWING HIGHLIGHTED GUIDANCE TEXT AFTER YOU READ IT AND REMOVE THE HIGHLIGHTING FROM THE DESIGN DECISION TEMPLATE. The following Design Decision is an example. Please follow the model below to communicate the design decisions appropriate to your customer and their requirements. See Section 3.1 for an example. 7.1.1 Design Decision 1 Description of the design decision 7.1.1.1. Option 1: Name Advantages: Advantage 1 Advantage 2 Drawbacks: Drawback 1 Drawback 2 7.1.1.2. Option 2: Name Advantages: Advantage 1 Advantage 2 Drawbacks: Drawback 1 Drawback 2 Further details should be included here. Also highlight any relevant requirements, assumptions and/or constraints that will impact this decision. Table 24. Option 1 Name or Option 2 Name Design Quality Option 1 Option 2 Comments Availability Both options improve availability, though Option 1 would guarantee a higher level. Manageability o Option 1 would be harder to maintain due to increased complexity. VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 39 of 79 Performance o o Both design options have no impact on performance Recoverability Both options improve recoverability Security o o Both design options have no impact on security Legend: = positive impact on quality; = negative impact on quality; o = no impact on quality Which option was selected and why? 7.2 vCenter Server Platform This section details the VMware vCenter Server proposed for the vSphere infrastructure design. Table 25. vCenter Server Logical Design Specifications Attribute Specification vCenter Server version 4.1 Physical or virtual system Virtual Number of CPUs Processor type Processor speed 2 VMware vCPU N/A Memory 4GB Number of NIC and ports 1/1 Number of disks and disk sizes 2: 12GB (C) and 40GB (D) Operating system and SP level Windows Server 2003 Enterprise R2 32-bit SP1
VMware vCenter Server, the heart of the vSphere infrastructure, is implemented on a virtual machine, as opposed to a standalone physical server. Virtualizing vCenter Server enables it to benefit from advanced features of vSphere, including VMware HA and vMotion. The exact vCenter Server build to be deployed will be selected closer to implementation and will be chosen based on the available stable and supported released versions at that time.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 40 of 79 7.3 vCenter Server Physical Design Specifications This section details the physical design specifications of the vCenter Server. Table 26. vCenter Server System Hardware Physical Design Specifications Attribute Specification Vendor and model VMware VM virtual hardware 7 Processor type VMware vCPU NIC vendor and model Number of ports/NIC x speed Network VMware Enhanced VMXNET 1 x Gigabit Ethernet Management network Local disk RAID level N/A
7.4 vCenter Server and Update Manager Databases This section details the specifications for the vCenter Server and Update Manager databases. Table 27. vCenter Server and Update Manager Databases Design Attribute Specification Vendor and version Microsoft SQL Server 2005 Authentication method SQL Server Authentication Recovery method Full Database autogrowth Enabled in 1MB increments Transaction log autogrowth In 10% increments; restricted to 2GB maximum size vCenter statistics level 3 Estimated database size 41.74GB
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 41 of 79 Setting Explanations Authentication method. Per the current database security policies, SQL Server Authentication, and not Windows Authentication, will be used to secure the vCenter Server databases. A SQL Server account with strong password will be created to support vCenter Server and vCenter Update Manager access to their respective databases. Recovery method. The full recovery method helps to ensure that no data is lost if there is database failure between backups. Because it maintains complete records of all changes to the database within the transaction logs, it is critical the database is backed up regularly which truncates (grooms) the logs. The DBA team schedules incremental nightly and full weekend backups of the vCenter and vCenter Update Manager databases. Database autogrowth. The vCenter Server database can expand on demand in 1MB increments with no restriction on its growth. Transaction log autogrowth. The transaction log is restricted to a maximum 2GBs to prevent filling up the log volume. Because the vCenter Server and vCenter Update Manager databases are backed up nightly (truncating the logs), the 2GB maximum should be more than required. vCenter statistics level. Level 3 gives more comprehensive vCenter statistics than the default setting. Estimated database size. 41.74GB was calculated using the vCenter Advanced Settings tool, assuming 24 hosts and 1,084 VMs at the above statistics level. The corporate database, Microsoft SQL Server 2005, will be used, as there is currently a trained team of DBAs supporting several physical database servers on this platform. For this initial vSphere infrastructure, this resource is leveraged and databases for both the vCenter Server and vCenter Update Manager are hosted on a separate, production physical database server system. Table 28. vCenter Server and Update Manager Database Names Attribute Specification vCenter database name VC01DB01 Update Manager database name VUM01DB01
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 42 of 79 The following table details the account to be created and its corresponding rights to each database. A strong password is assigned to this account and documented following established password storage procedures. Table 29. SQL Server Database Accounts Account Name Database Rights vc01 VC01DB01 dbowner vc01 VUM01DB01 dbowner vc01 msdb dbowner* * dbowner rights to the msdb database can and will be revoked following vCenter installation and creation of the vCenter Server and vCenter Update Manager databases.
Table 30. ODBC System DSN Database ODBC System DSN ODBC Client Database Server SQL Account vCenter Server VC01DB01 SQL Native Client SQLPROD08 vc01 Update Manager VUM01DB01 SQL Native Client SQLPROD08 vc01
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 43 of 79 DELETE THE FOLLOWING HIGHLIGHTED TEXT AFTER YOU READ IT This sample is based on Enterprise Plus licensing. Be sure to educate your customer to the capabilities of each feature. The actual licensing that your customer has purchased will be based on their business requirements. You will need to update this section accordingly. 7.5 Licenses For this initial vSphere infrastructure, the Enterprise Plus vSphere license edition is used. This edition provides the following licensed features: Up to 8 physical cores per CPU and no physical memory limit per ESX/ESXi host Update Manager vStorage APIs for Data Protection High Availability (HA) Thin provisioning Data Recovery vMotion Hot add virtual hardware support Fault Tolerance (FT) vShield Zones Storage vMotion DRS DPM vNetwork Distributed Switches Host Profiles vStorage APIs for Multipathing Virtual Serial Port Concentrator vStorage APIs for Array Integration Network I/O Control Storage I/O Control The vCenter Server will be configured with the issued 25-character license keys and automatically installs the appropriate license on the ESX/ESXi hosts as they are added to inventory. The License Reporting Manager is used to view and generate reports on a quarterly basis for all license keys for vSphere 4.1 products in the virtual IT infrastructure.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 44 of 79 8. vSphere Infrastructure Security 8.1 Overview Security is absolutely critical in this environment, and any security vulnerability or risk exposed by the new vSphere infrastructure would have a negative impact on future adoption of virtualization technology. To protect the business, existing security policies and procedures were considered and leveraged. Microsoft Active Directory users and groups were used to govern access. To reduce confusion that could undermine security, wherever possible the existing administration teams are granted the same access to the vSphere environment as they currently have to the physical infrastructure. For example, the physical network team is granted access to facilitate their responsibility of the virtual network infrastructure. However, end users will continue to access the virtual machines through the guest OS or application mechanisms and will not have access through VMware vSphere components or the vSphere Client directly. No access is granted that is not required to perform a specific, authorized job function. 8.2 Host Security Chosen in part for its limited management console functionality, ESXi is configured with a strong root password stored following the corporate password guidelines. ESXi lockdown mode is also enabled to prevent root access to the hosts over the network, and appropriate security policies and procedures are created and enforced to govern the systems. Because ESXi cannot be accessed over the network, sophisticated host-based firewall configurations are not required. A new team; ESX Admins will be created with a supporting Active Directory group. This, and only this, team has overall responsibility and access to the ESX/ESXi hosts. 8.3 vCenter and Virtual Machine Security Access to perform systems administration of the virtual machines is divided among the same teams currently responsible for the physical instances of the systems, leveraging the existing Active Directory implementation. Folders within vCenter Virtual Machine and Templates inventory are created to simplify assignment of these permissions to the appropriate virtual machines. Folders are created for each different system classification (such as HR, Finance, IT) and used to contain their respective virtual machines. Permissions are applied to these folders and mapped to the appropriate Active Directory group responsible for the management of the virtual machines they contain. A new team will also be created: Enterprise vSphere Administration are created with a supporting Active Directory group. This, and only this, team has overall and overarching responsibility and access to the entire vSphere infrastructure. The existing physical network and storage administration teams are also granted access, with their job responsibilities extended into the virtual infrastructure. Leveraging security capabilities of vSphere, the network and storage teams are granted access only to virtual networking and storage, respectively; that is, only what is required to perform their specific job responsibilities. Neither team has access to virtual machines, the datacenter, clusters, or any other aspect of the vSphere infrastructure. Similarly, the virtual machine admin teams do not have access to the configuration of network, storage, or any other aspects of the vSphere infrastructure, only the virtual machines they are responsible for. Only the new Enterprise vSphere Administration team has access to the entire infrastructure. As stated previously, no end users are granted access to the vSphere Infrastructure directly. They continue to leverage guest OS and application interfaces for connectivity. Appendix G contains detailed information regarding the assignment of VMware vCenter Roles to Microsoft Active Directory Groups used to secure the infrastructure. VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 45 of 79 8.4 vSphere Port Requirements Appendix H contains detailed information regarding which ports to open for communication between vSphere-related services. 8.5 Lockdown Mode and Troubleshooting Services To remain compliant with the organizations security regulations, Lockdown Mode is enabled. All configuration changes to the vSphere environment are made accessing the vCenter Server with lockdown mode configured. Lockdown mode restricts access to host services on the ESXi server, but does not affect the availability of these services. Note If the ESXi host loses access to vCenter Server while running in Total Lockdown Mode, ESXi must be reinstalled to regain access to the host.
Table 31. Lockdown Mode Configurations Service Lockdown Mode Total Lockdown Mode Lockdown On On Local Tech Support Mode Off Off Remote Tech Support Mode Off Off Direct Console User Interface On Off
ESXi features three types of troubleshooting services; Local Tech Support Mode (TSM), Remote Tech Support Mode Service (SSH) and Direct Console User Interface Service (DCUI). In accordance with the organizations security policy TSM and SSH are enabled only when required. DCUI remains enabled to conform to design specifications. Lockdown and Troubleshooting Settings Explanation Local Tech Support Mode (TSM). Usable with physical access to the server console and root privileges.TSM provides a command-line interface to diagnose and repair VMware ESXi hosts. Support Mode should only be used at the request of VMware technical support. Remote Tech Support Service (SSH). Similar to TSM, but uses the secure protocol SSH for remote access to the server. Requires root privileges and provides a command-line interface to diagnose and repair VMware ESXi hosts. Support Mode should only be used at the request of VMware technical support. Direct Console user Interface Service (DCUI). Usable with physical access to the server and root privileges. The DCUI is used for basic configuration of the ESXi host. When running in Lockdown mode, you can log in locally to the direct console user interface as the root user and disable Lockdown Mode. You can then troubleshoot the issue using a direct connection to the vSphere Client or by enabling Tech Support Mode.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 46 of 79 9. vSphere Infrastructure Monitoring 9.1 Overview Because the uptime and health of the entire technology infrastructure is paramount, the current environment already has an implementation of an enterprise monitoring system. Capable of both SNMP-based and managed system-specific monitoring, such as leveraging Windows Events and Performance Counters, the system monitors all network, server, and storage systems, processing any events through a sophisticated event correlation engine to determine the appropriate response and notifications to send. The new vSphere infrastructure integrates into the monitoring system through the appropriately configured events and alerts. 9.2 Server, Network, and SAN Infrastructure Monitoring All of the physical systems, including the network and SAN, will continue to be monitored directly by the enterprise monitoring system which is configured to incorporate additional infrastructure required to support vSphere. The new physical servers purchased to run VMware ESX/ESXi are outfitted with IPMI Baseboard Management Controllers (BMC) used by the enterprise monitoring system to monitor system hardware status (such as processor temperature, fan speed, and the like). In the future, vSphere Distributed Power Management (DPM) will be considered, and can use these IPMI BMCs to automatically power ESX/ESXi Hosts on and off based on demand to help further power and cooling savings. 9.3 vSphere Monitoring Leveraging the event monitoring and alarm system in vSphere, vCenter Server is configured to monitor the health and performance of all critical virtual infrastructure components, including the ESX/ESXi hosts, the clusters, VMware HA, Fault Tolerance, virtual machine operations such as vMotion, and the health of the vCenter Server itself. The events and conditions to be monitored and configured to alert are detailed in Appendix I. Upon the triggering of an alert, vCenter Server is configured to send SNMP traps to the enterprise management systems SNMP receiver. Although the same system is primarily responsible for event correlation and email alerting across the enterprise, vCenter Server is also configured to send email alerts for all triggered events to the vSphere Enterprise Administration group. The vSphere Enterprise Administrators group is also responsible for routinely reviewing and managing the health and system logs generated by the ESX/ESXi hosts, vCenter Server, and the virtual machines. These logs will be groomed and archived following corporate log retention policies and procedures. The following are monitored on a consistent basis to verify the health of the virtual infrastructure: VMware ESX/ESXi host hardware status vCenter Server services status VMware HA Healthcheck and operational status Cluster operational status Storage performance statistics Network performance statistics Network I/O control status Storage I/O control status Memory utilization report VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 47 of 79 9.4 Virtual Machine Monitoring The current enterprise management system provides monitoring of the 1,000 systems to be virtualized and continues performing this important task after the systems are converted to vSphere VMs. After reviewing the current monitoring configuration, it was determined that only minor changes are necessary to facilitate monitoring the systems after they are virtualized. The monitoring system primarily requires network connectivity to the virtual machines which is not impacted by the conversion, as their IP addresses and host names are not changing. However, the mechanism that monitors the performance of Windows virtual machines utilizes Windows Performance Monitor (Perfmon) counters and must be reconfigured to use new, virtualization- specific Windows Perfmon counters provided by VMware Tools. These counters, unlike their counterparts for physical components, are tuned for accurate assessment of virtualized Windows performance. Appendix I provides detailed monitoring system configuration information, including SNMP and SMTP settings, the list of alarms and events to leverage, and the Windows Performance Monitor counters to use by the enterprise monitoring system to monitor virtual machines.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 48 of 79 10. vSphere Infrastructure Patch/Version Management 10.1 Overview Maintaining an up-to-date IT infrastructure is critical. The health, performance, and security of the entire business depends on the health, performance, and security of its supporting technology. Maintaining an up-to-date infrastructure can be a daunting task for IT administrators, but if not performed dependably and routinely, the infrastructure is at risk. The VMware vCenter Update Manager enterprise patch automation tool is implemented as part of the new vSphere infrastructure to keep the vSphere ESX/ESXi hosts and virtual machines VMware Tools up-to-date. Update Manager can provision, patch, and upgrade third-party modules such as EMC's PowerPath multipathing software. Administrators can also evaluate patches/updates to VMware vCenter Server and the vSphere Client and update those manually as required. 10.2 vCenter Update Manager VMware vCenter Update Manager is an automated patch management solution that applies patches and updates to VMware ESX/ESXi hosts, Microsoft Windows virtual machines, and select Linux virtual machines. vCenter Update Manager can also update VMware Tools and VMware virtual hardware in virtual machines. In addition to securing the datacenter against vulnerabilities and reducing or eliminating downtime related to host patching, automated ESX/ESXi host updates provide a common, installed version for hosts. Although VMware Fault Tolerance includes a versioning-control mechanism that allows the Primary and Secondary VMs to run on FT-compatible hosts at different but compatible patch levels, maintaining common versions among the hosts is the best practice. This is vital for the health of VMware Fault Tolerance, which requires the same build to be installed on all hosts supporting FT-protected VMs. vCenter Update Manager will be installed on the same system as vCenter Server and configured to patch/upgrade the ESX/ESXi hosts and VMware Tools installed within the virtual machines. It will, however, not be used to automatically update virtual machine hardware. Virtual machine hardware updates are evaluated and performed manually as needed. For this initial deployment, Update Manager is not leveraged to update virtual machines guest operating systems or applications, as there is already a robust mechanism in place for such patching. Because Update Manager is not used to patch VMs, the Update Manager Server can reside on the same system as the vCenter Server for simplicity. In the future, if Update Manager is used to update VMs, the Update Manager Server should be run on a separate dedicated system for performance reasons, so as not to overburden the vCenter Server.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 49 of 79 Like vCenter, a dedicated database for Update Manager will be created on one of the database servers also housing the database for vCenter Server using the vCenter Update Manager Sizing Estimator from vmware.com, and using the following assumptions: No remediation of VMs Remediate ESXi 4.1+ hosts (no ESX 4.1+) 1 concurrent ESXi host upgrades 24 hosts Patch scan frequency for hosts: 4 per month Upgrade scan frequency for hosts: 1 per month Patching the vCenter Update Manager database is estimated to require 870 MB of storage for the first year and up to 2GBs of patch and 2GBs of temporary disk storage space on the Update Manager Server system for patches. Table 32. Estimated Update Manager Storage Requirements Estimated Update Manager Database Size (first year) Estimated Patch Storage Required Estimated Temp Storage Required 870 MB 2GB 2GB
Table 33. vCenter Update Manager Specifications Attribute Specification Patch download sources Select Download ESX 4.1 patches Unselect Download ESX 3 patches, Download Linux VM patches, Download Windows VM patches Shared repository D:\vCenterUpdateManager\vSpherePatches Proxy settings None Patch download schedule Every Sunday at 12:00AM EST Email notification vsphereadmins@domain.local Update Manager baselines to leverage Critical and non-critical ESX/ESXi host patches VMware Tools upgrade to match host Virtual machine settings Select Snapshot virtual machines before remediation to enable rollback Select Dont delete snapshots ESX/ESXi host settings Host maintenance mode failure: Retry Retry interval: 30 Minutes Number of retries: 3 vApp settings Select Enable smart reboot after remediation VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 50 of 79 Setting Explanations Patch download sources. Which patches to download. Shared repository. The vCenter Update Manager Server was configured with a data disk to use for storing patches at this location. Proxy settings. Settings for a proxy server if one is used to access the internet from the datacenter. Patch download schedule. The time and frequency to download new patches. Email notification. Who Update Manager automatically notifies when new patches are downloaded. Update Manager baselines to leverage. Update Manager baselines define a level of patches and updates to monitor for and download. Virtual machine settings. Initially, Update Manager will not be used to update virtual machines. However, this setting will be configured per best practices to prepare for the event that when VM patching is activated, a snapshot will be taken of each virtual machine prior to performing any remediation operations. This will enable rollback of patches that are applied by vCenter Update Manager, if necessary. These snapshots will not be automatically deleted by Update Manager. Members of the vSphere Administration group will delete the snapshots after determining that the patches have been successfully applied and are functioning correctly. ESX/ESXi host settings. Update Manager places a host into maintenance mode before applying patches. Maintenance mode automatically triggers the migration of any VMs running on the host to other hosts in the cluster to avoid VM downtime. If Update Manager and vCenter encounter problems putting a host into maintenance mode, this setting specifies what to do and how many times within the specified interval between attempts before abandoning the patch application to a particular host. vApp settings. vApps are logical groups of VMs. This setting uses the start order of VMs as defined with a vApp when powering on VMs. vApps often require powering on virtual machines in a specific order due to dependencies, and this is configured within the vApps properties. 10.3 vCenter Server and vSphere Client Updates Administrators routinely check for and evaluate new vCenter Server and vSphere Client updates. These are installed in a timely fashion following release and proper testing. VMware vSphere Client updates should be manually installed whenever vCenter Server is updated. Using conflicting versions of the vSphere Client and vCenter Server can cause unexpected results. vSphere Client automatically checks for and downloads an update if it exists when connecting to an updated vSphere Server.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 51 of 79 11. Backup/Restore Considerations 11.1 Hosts Host Profiles are created and used to restore ESX/ESXi hosts. Backing up hosts is not a necessary practice because a standard installation is relatively trivial and takes only minutes from start to finish. Administrators maintain accurate documentation of the ESX/ESXi networking and storage configurations to use for reference after a restore. 11.2 Virtual Machines There currently is a robust enterprise backup system that is used to back up each physical system. The plan is to continue using this method to back up virtual machines. Restoring virtual machine guest operating systems, applications, and associated data follows the same method as for physical machines. The current production backup strategy calls for incremental backups nightly, full backups on Sundays, and full backups at month-end and year-end. Test, development, and QA VMs are backed up on an as-needed basis. In the future, virtualization-specific methods for backing up and restoring VMs will be considered.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 52 of 79 12. Design Assumptions 12.1 Hardware Hardware deployment must meet technical requirements for each product. If the hardware used deviates from the recommended hardware (see Section 13, Reference Documents), there must be re-qualification from the development team to make sure that this hardware supports all VMware products used in the deployment. The technical assumptions for this design are listed below. Table 34. Sources of Technical Assumptions for this Design Element Reference ESX/ESXi and vCenter Server ESX/ESXi and vCenter Installation Guide ESX/ESXi Configuration Guide Basic System Administration Guide vSphere 4.1 Configuration Maximums Guide ESX/ESXi host hardware vSphere Hardware Compatibility Lists ESX/ESXi I/O adapters vSphere Hardware Compatibility Lists ESX/ESXi SAN compatibility Fibre Channel SAN Configuration Guide iSCSI SAN Configuration Guide vMotion, HA, fault tolerance vSphere 4.1 Availability Guide
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 53 of 79 12.2 External Dependencies External dependencies address other systems or technologies that depend on or could be affected by the vSphere infrastructure. External dependencies are different assumptions in that they clearly identify dependent factors and the consequent implications. Table 35. VMware Infrastructure External Dependencies Item Requirements Active Directory Active Directory is required to implement and operate the vSphere Infrastructure. DNS DNS must be configured for connectivity between vCenter Server, Active Directory, VMware ESX/ESXi, and virtual machines. Network Network congestion or failure prevents vMotion from migrating virtual machine and affects the ability of vCenter Server to manage VMware ESX/ESXi hosts. It can also negatively impact HA and FT. Storage Area Network Stability and performance of the SAN affects the virtual machines. Time synchronization Accurate time keeping and time synchronization is critical for a healthy vSphere infrastructure. All components including ESX/ESXi hosts, vCenter Server, the SAN, physical network infrastructure, and virtual machine guest operating systems must have accurate time keeping. This is especially critical for virtual machines protected by FT. Staff Properly trained IT staff is critical for the correct implementation, operation, support, and enhancement of the vSphere infrastructure. Policies and procedures The policies and procedures governing the use of information technology must be revised to properly incorporate the unique properties and capabilities of virtualization as implemented through this design.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 54 of 79 13. Reference Documents 13.1 Supplemental White Papers and Presentations ESX and vCenter Server Installation Guide for ESX 4.1, vCenter Server 4.1 ESXi Installable and vCenter Server Setup Guide for ESXi 4.1 Installable, vCenter Server 4.1 ESXi Embedded and vCenter Server Setup Guide for ESXi 4.1 Embedded, vCenter Server 4.1 vSphere Datacenter Administration Guide for ESX 4.1, ESXi 4.1, vCenter Server 4.1 http://www.vmware.com/pdf/vsphere4/r41/vsp_41_dc_admin_guide.pdf vSphere Resource Management Guide for ESX 4.1, ESXi 4.1, vCenter Server 4.1 http://www.vmware.com/pdf/vsphere4/r41/vsp_41_resource_mgmt.pdf VMware ESXi and ESX Info Center http://www.vmware.com/products/vsphere/esxi-and-esx/upgrade.html VMware Enterprise Infrastructure Support Life Cycle Policy http://www.vmware.com/support/policies/lifecycle/enterprise-infrastructure/index.html vSphere Management Assistant Guide for vSphere 4.1 http://www.vmware.com/support/developer/vima/vma41/doc/vma_41_guide.pdf VMware Data Recovery Administration Guide for Data Recovery 1.2 http://www.vmware.com/pdf/vdr_12_admin.pdf vSphere Compatibility Matrixes http://www.vmware.com/pdf/vsphere4/r40/vsp_compatibility_matrix.pdf Configuration Maximums for VMware vSphere 4.1 http://www.vmware.com/pdf/vsphere4/r41/vsp_41_config_max.pdf iSCSI SAN Configuration Guide for ESX 4.1, ESXi 4.1, vCenter Server 4.1 http://www.vmware.com/pdf/vsphere4/r41/vsp_41_iscsi_san_cfg.pdf Fibre Channel SAN Configuration Guide for ESX 4.1, ESXi 4.1, vCenter Server 4.1 http://www.vmware.com/pdf/vsphere4/r41/vsp_41_san_cfg.pdf Setup for Failover Clustering and Microsoft Cluster Service for ESX 4.1, ESXi 4.1, vCenter Server 4.1 http://www.vmware.com/pdf/vsphere4/r41/vsp_41_mscs.pdf VMware Network I/O Control: Architecture, Performance and Best Practices for VMware vSphere 4.1 http://www.vmware.com/files/pdf/techpaper/VMW_Netioc_BestPractices.pdf vSphere Command-Line Interface Installation and Scripting Guide for ESX 4.1, ESXi 4.1, vCenter Server 4.1 http://www.vmware.com/pdf/vsphere4/r41/vsp4_41_vcli_inst_script.pdf VMware vCenter Converter Installation and Administration Guide for vCenter Converter 4.2 http://www.vmware.com/pdf/vsp_vcc_42_admin_guide.pdf vSphere Management Assistant Guide for vSphere 4.1 http://www.vmware.com/support/developer/vima/vma41/doc/vma_41_guide.pdf VMware Compatibility Guides http://www.vmware.com/resources/compatibility/search.php VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 55 of 79 Introduction to VMware vSphere for ESX 4.0, ESXi 4.0, vCenter Server 4.0 http://www.vmware.com/pdf/vsphere4/r40/vsp_40_intro_vs.pdf vShield Zones Administration Guide for vShield Zones 1.0 http://www.vmware.com/pdf/vsz_10_admin.pdf Whats New in VMware vSphere 4: Performance Enhancements: http://www.vmware.com/files/pdf/VMW_09Q1_WP_vSpherePerformance_P13_R1.pdf Whats New in VMware vSphere 4:Virtual Networking: http://www.vmware.com/files/pdf/VMW_09Q1_WP_vSphereNetworking_P8_R1.pdf What Is New in VMware vSphere 4: Storage: http://www.vmware.com/files/pdf/VMW_09Q1_WP_vSphereStorage_P10_R1.pdf Network Segmentation in Virtualized Environments http://www.vmware.com/files/pdf/network_segmentation.pdf Protecting Mission-Critical Workloads with VMware Fault Tolerance: http://www.vmware.com/files/pdf/resources/ft_virtualization_wp.pdf VMware ESX 3 802.1Q VLAN Solutions: http://www.vmware.com/pdf/esx3_vlan_wp.pdf CLARiiON Integration with VMware ESX: http://www.vmware.com/pdf/clariion_wp_eng.pdf Recommendations for Aligning VMFS Partitions: http://www.vmware.com/pdf/esx3_partition_align.pdf Security Design of the VMware Infrastructure 3 Architecture: http://www.vmware.com/pdf/vi3_security_architecture_wp.pdf Making Your Business Disaster Ready with VMware Infrastructure: http://www.vmware.com/pdf/disaster_recovery.pdf Automating High Availability (HA) Services with VMware HA: http://www.vmware.com/pdf/vmware_ha_wp.pdf ESX 4 Patch Management Guide: http://www.vmware.com/pdf/vsphere4/r40/vsp_40_esxupdate.pdf Best Practices for Patching ESX: http://www.vmware.com/resources/techresources/1075 Microsoft Exchange Server 2007 Performance on VMware vSphere 4 http://www.vmware.com/files/pdf/perf_vsphere_exchange-per-scaling.pdf Improving Scalability for Citrix Presentation Server: http://www.vmware.com/pdf/esx_citrix_scalability.pdf Managing VMware VirtualCenter Roles and Permissions: http://www.vmware.com/resources/techresources/826 VMware vCenter Update Manager Performance and Best Practices http://www.vmware.com/pdf/Perf_UpdateManager40_Best-Practices.pdf
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 56 of 79 13.2 Supplemental KB Articles vMotion CPU Compatibility Requirements for Intel Processors: http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1991 vMotion CPU Compatibility Requirements for AMD Processors: http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1992 vMotion CPU Compatibility - Migrations Prevented Due to CPU Mismatch - How to Override Masks http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&extern alId=1993&sliceId=1&docTypeID=DT_KB_1_1&dialogID=23256056&stateId=0 0 2325069 Installing ESX.1 and vCenter 4. 41 Best Practices: http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&extern alId=1009080&sliceId=2&docTypeID=DT_KB_1_1&dialogID=23256161&stateId=0%200%20 23250853 VMware High Availability Slot Calculation: http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&extern alId=1010594&sliceId=1&docTypeID=DT_KB_1_1&dialogID=23256209&stateId=0%200%20 23250906 VMware vSphere 4.1 Software Compatibility Matrix: http://partnerweb.vmware.com/comp_guide/docs/vSphere_Comp_Matrix.pdf Processors and Guest Operating Systems that Support VMware Fault Tolerance: http://kb.vmware.com/kb/1008027 VMware Infrastructure Architecture Overview: http://www.vmware.com/pdf/vi_architecture_wp.pdf Virtualization Overview: http://www.vmware.com/pdf/virtualization.pdf Whats New in VMware vSphere 4:Virtual Networking: http://www.vmware.com/files/pdf/VMW_09Q1_WP_vSphereNetworking_P8_R1.pdf What Is New in VMware vSphere 4: Storage: http://www.vmware.com/files/pdf/VMW_09Q1_WP_vSphereStorage_P10_R1.pdf Network Throughput in a VMware Infrastructure: http://www.vmware.com/pdf/esx_network_planning.pdf Network Segmentation in Virtualized Environments http://www.vmware.com/files/pdf/network_segmentation.pdf The vSphere Availability Guide: http://www.vmware.com/pdf/vsphere4/r40/vsp_40_availability.pdf Protecting Mission-Critical Workloads with VMware Fault Tolerance: http://www.vmware.com/resources/techresources/1094 VMware ESX 3 802.1Q VLAN Solutions: http://www.vmware.com/pdf/esx3_vlan_wp.pdf CLARiiON Integration with VMware ESX: http://www.vmware.com/pdf/clariion_wp_eng.pdf Recommendations for Aligning VMFS Partitions: http://www.vmware.com/pdf/esx3_partition_align.pdf VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 57 of 79 Security Design of the VMware Infrastructure 3 Architecture: http://www.vmware.com/pdf/vi3_security_architecture_wp.pdf Making Your Business Disaster Ready with VMware Infrastructure: http://www.vmware.com/pdf/disaster_recovery.pdf Automating High Availability (HA) Services with VMware HA: http://www.vmware.com/pdf/vmware_ha_wp.pdf ESX 4 Patch Management Guide: http://www.vmware.com/pdf/vsphere4/r40/vsp_40_esxupdate.pdf Best Practices for Patching ESX: http://www.vmware.com/resources/techresources/1075 Microsoft Exchange Server 2007 Performance on VMware vSphere 4 http://www.vmware.com/files/pdf/perf_vsphere_exchange-per-scaling.pdf Improving Scalability for Citrix Presentation Server: http://www.vmware.com/pdf/esx_citrix_scalability.pdf Managing VMware VirtualCenter Roles and Permissions: http://www.vmware.com/resources/techresources/826 VMware vCenter Update Manager Performance and Best Practices http://www.vmware.com/pdf/Perf_UpdateManager40_Best-Practices.pdf 13.3 Supplemental KB Articles vMotion CPU Compatibility Requirements for Intel Processors: http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1991 vMotion CPU Compatibility Requirements for AMD Processors: http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1992 vMotion CPU Compatibility - Migrations Prevented Due to CPU Mismatch - How to Override Masks http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&extern alId=1993&sliceId=1&docTypeID=DT_KB_1_1&dialogID=23256056&stateId=0 0 2325069 Installing ESX 4.1 and vCenter 4.1 best practices: http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&extern alId=1009080&sliceId=2&docTypeID=DT_KB_1_1&dialogID=23256161&stateId=0%200%20 23250853 VMware High Availability slot calculation: http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&extern alId=1010594&sliceId=1&docTypeID=DT_KB_1_1&dialogID=23256209&stateId=0%200%20 23250906 VMware vSphere 4.1 Software Compatibility Matrix: http://partnerweb.vmware.com/comp_guide/docs/vSphere_Comp_Matrix.pdf Processors and Guest Operating Systems that Support VMware Fault Tolerance: http://kb.vmware.com/kb/1008027 VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 58 of 79 Appendix A ESX/ESXi Host Estimation To determine the number of hosts required to consolidate the existing datacenters 1,100 physical x86 servers, the performance and utilization of the existing servers was analyzed using VMware Capacity Planner for 30 days. The analysis captured the resource utilization for each system, including average and peak CPU and RAM utilization. Out of the servers analyzed, 100 were disqualified from the consolidation project for the following reasons: Servers are already planned for decommission and there is no need to virtualize Incomplete performance and utilization data; such servers to be deferred for further analysis Servers use specialized hardware that cannot be virtualized Note ESX 4.1 supports USB device passthrough from an ESX or ESXi to a virtual machine. A total of 1,000 candidates were selected for this first virtualization initiative. Over the sampling period, the metrics in the following tables were observed. Table 36. CPU Resource Requirements Metric Amount Average number of CPUs per physical system 2 Average CPU MHz 2800 MHz Average normalized CPU per physical system 5663 MHz Average CPU utilization per physical system 6.5% (368.01 MHz) Average peak CPU utilization per physical system 9% (509.67 MHz) Total CPU resources required for 1,000 VMs at peak 509,670 MHz
Table 37. RAM Resource Requirements Metric Amount Average amount of RAM per physical system 1024 MB Average memory utilization 62% (634.88 MB) Average peak memory utilization 70% (716.80 MB) Total RAM required for 1000 VMs at peak before memory sharing 716,800 MB Anticipated memory sharing benefit when virtualized 50% Total RAM required for 1,000 VMs at peak with memory sharing 358,400 MB VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 59 of 79 For estimating capacity, the target host platform has the specifications proposed in the following tables. Table 38. Proposed ESX/ESXi Host CPU Logical Design Specifications Attribute Specification Number of CPUs (sockets) per host 4 Number of cores per CPU 4 MHz per CPU core 2,400 Total CPU MHz per CPU 9,600 Total CPU MHz per host 38,400 Proposed maximum host CPU utilization 80% Available CPU MHz per host 30,720 MHz
Table 39. Proposed ESX/ESXi Host RAM Logical Design Specifications Attribute Specification Total RAM per host 32,768 MB (32 GB) Proposed maximum host RAM utilization 80% Available RAM per host 26,214 MB
Estimation Assumptions Hosts sized for peak utilization levels rather than average utilization. This is to support all systems running at their observed peak resource levels simultaneously. CPU and memory utilization for each host capped at 80% (allow 20% for overhead and breathing room). Memory sharing: 50% (achieved through running the same guest OS, Windows Server 2003 Standard R2, across 90% of all VMs).
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 60 of 79 The following formula was used in calculating estimated required host capacity to support the peak CPU utilization of the anticipated VM workloads:
Total CPU required for total VMs at peak =#of ESX/ESXi Hosts Required Available CPU per ESX/ESXi Host Using this formula, the following estimated required host capacity was calculated for the planned vSphere infrastructure:
The following formula was used in calculating the number of hosts required to support anticipated at peak RAM utilization:
Total RAM required for total VMs at peak = #of ESX/ESXi Hosts Required Available RAM per ESX/ESXi Host
Using this formula, the following estimated required host capacity was calculated for the planned vSphere infrastructure: 358,400 MB (Total RAM) = 13.67 ESXi Hosts 26,214 MB (RAM per Host)
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 61 of 79 From a CPU workload perspective, 17 VMware ESX/ESXi hosts are needed, but from a memory workload perspective, only 14 hosts are needed. The higher value is used because that is the limiting factor. This provides substantial consolidation ratios as shown in the following table. Table 40. VMware vSphere Consolidation Ratios # of Virtualization Candidates # of ESX/ESXi Hosts Required Consolidation Ratio: VMs per Host Consolidation Ratio: VMs per Core* Max Host CPU/ RAM Utilization 1,000 17 58.82 3.68 80% * Each VM has one vCPU
In actuality, because 1,000 VMs can be supported by 16.6 hosts, the true consolidation ratio is 60.27, which means that through extrapolation with 17 hosts, the infrastructure should be able to support not just 1,000 VMs, but 1,024 VMs. VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 62 of 79 Appendix B ESX/ESXi Host PCI Configuration The selected host platform has 11 PCIe expansion slots. Card locations were staggered to simplify system cabling, leaving five (5) available PCIe slots for future expansion. Table 41. ESX/ESXi Host PCIe Slot Assignments Slot Number Card 1 Dual port PCIe-4x Fibre Channel HBA 2 Empty 3 Dual port PCIe-4x Fibre Channel HBA 4 Empty 5 Dual port PCIe-4x Gigabit Ethernet adapter 6 Empty 7 Dual port PCIe-4x Gigabit Ethernet adapter 8 Empty 9 Dual port PCIe-4x Gigabit Ethernet adapter 10 Empty 11 Dual port PCIe-4x Gigabit Ethernet adapter VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 63 of 79 Appendix C Hardware BIOS Settings The default hardware BIOS settings on servers may not always be the best choice for optimal performance. Review the BIOS settings with a VMware virtualized environment in mind. This section lists some of the BIOS settings for consideration. The latest version of the BIOS available for your system is running. BIOS is set to enable all populated sockets and all cores in each socket. Turbo Mode is enabled if your processors support it. Hyperthreading is enabled in the BIOS. Some NUMA-capable systems provide an option to disable NUMA by enabling node interleaving. In most cases, the best performance is achieved by disabling node interleaving Hardware-assisted virtualization features (VT-x, AMD-V, EPT, RVI) are enabled in the BIOS. Disable C1E halt state in the BIOS. (See note following regarding performance considerations versus power considerations.) Disable any other power-saving mode in the BIOS. (See note following regarding performance considerations versus power considerations.) Disable unneeded devices, such as serial and USB ports, from the BIOS Notes ESX 4.0 supports Enhanced Intel SpeedStep and Enhanced AMD PowerNow! CPU power management technologies that can save power when a host is not fully utilized. However, because these and other power-saving technologies can reduce performance in some situations, consider disabling them when performance considerations outweigh power considerations. Because of the large number of different server models and configurations, any list of BIOS options is always likely to be incomplete. After updating the BIOS, make sure your BIOS settings are as you wanted them. After these changes are made, some systems may need a complete power down and restart before the changes take effect.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 65 of 79 Appendix E Storage Volume Specifications Table 43. VMFS Volumes Datastore Name LUN ID Purpose prod_san01_00 0 Production prod_san01_01 1 Production prod_san01_02 2 Production prod_san01_03 3 Production ~ ~ ~ prod_san01_46 46 Production test_san01_47 47 Test test_san01_48 48 Test
Table 44. NFS Volumes NFS Volume Name Server Path nfsvol01 filer01.domain.local /nfs/vsphere/nfsvol01 nfsvol02 filer01.domain.local /nfs/vsphere/nfsvol02 images filer03.domain.local /nfs/shared/images
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 66 of 79 Appendix F LUN Sizing Recommendations When sizing a LUN for a VMware Datastore, first determine how many active virtual machines, (VMs), to allocate per LUN. For read/write performance, it does not matter if your VMDK is on a 100GB LUN or 2TB LUN. It is the total I/O load on the LUN that matters. Below is a general calculation you can use to determine the LUN size best suited for your environment. Suppose for example you determined that you are using 10 VMDKs per LUN on average, then calculate the average disk size. For this example, 50GB is an average size for the VMDK file: Drive C:\ 20GB OS and patches Drive D:\ 30GB Application disk TOTAL 50 GB VMDK file Multiplying 50GB by 10 VMs comes to 500GB per LUN to store 10 VMDKs. 10 * 50 = 500 Swap Space Now factor in the needed spare room for VM swap space and for snapshots. For this example, on average VMs are configured with 3GB of RAM. This requires a 3GB swap space for each VM. Ten VMs would require a total 30GB of swap space. 10 * 3 = 30.0 Snapshots 15% is a good figure to use for snapshot overhead. The retention policy for active snapshots determines the overall space required. Best practice recommends that the retention policy for active snapshots be as short as possible. If the retention policy exceeds two days, then store the snap shots on a second tier datastore. 15% of 500GB = 75.0 Free Space Factor in 15% for free space: 15% of 500GB = 75.0
To summarize and build the formula: 10 VMs x 50GB (avg. disk size) = 500.0 GB (total disk size) + 10 VMs x 3GB = 30.0 GB (VM swap) + 15% of 500GB = 75.0 GB (snapshots) + 15% of 500GB = 75.0 GB (free space) Total 680.00 Round Up 700.00 GB
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 67 of 79 vSphere 4.1 offers enhanced visibility into storage throughput, and latency of hosts and virtual machines, and aids in troubleshooting storage performance issues. NFS statistics are now available in vCenter Server performance charts, as well as ESXTOP. Storage I/O provides quality of service in the form of I/O shares and limits. Using Storage I/O Control, vSphere administrators can make sure that the most important virtual machines get adequate I/O resources even in times of congestion. Refer to the vSphere Resource Management Guide and vSphere Datacenter Administration Guide for more detail.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 68 of 79 Appendix G Security Configuration Table 45. vSphere Roles and Permissions vSphere Role Name Corresponding AD Groups Enabled vSphere Privileges vCenter Inventory Level for Permissions Description Enterprise vSphere Administrators* vSphere-Admins* All Datacenter and all child objects Administrative rights to the entire vSphere infrastructure vSphere Network Administrators* NET-Admins Network and all child privileges Network and all network child objects ONLY Administrative rights to all vSphere network components vSphere Storage Administrators* SAN-Admins Datastore and all child privileges Storage Views and all child privileges Datastores and all datastore child objects ONLY Administrative rights to all vSphere storage components Virtual Machine Administrators* FINSYS-Admins Virtual Machine and all child privileges FINANCE folder and the VMs it contains ONLY Administrative rights to Finance Virtual Machines ONLY Virtual Machine Administrators* HRSYS-Admins Virtual Machine and all child privileges HR folder and the VMs it contains ONLY Administrative rights to HR Virtual Machines ONLY Virtual Machine Administrators* DEVSYS-Admins Virtual Machine and all child privileges DEVELOPMENT folder and the VMs it contains ONLY Administrative rights to Development Virtual Machines ONLY Virtual Machine Administrators* QASYS-Admins Virtual Machine and all child privileges QA folder and the VMs it contains ONLY Administrative rights to QA Virtual Machines ONLY Virtual Machine Administrators* ITSYS-Admins Virtual Machine and all child privileges IT folder and the VMs it contains ONLY Administrative rights to IT Virtual Machines ONLY * New. Must be created.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 69 of 79 Table 46. vCenter Virtual Machine and Template Inventory Folders to be used to Secure VMs Folder VM Type Associated Admin AD Group FINANCE Financial VMs FINSYS-Admins HR HR VMs HR-Admins DEVELOPMENT Development VMs DEV-Admins QA QA VMs QA-Admins IT IT VMs IT-Admins
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 70 of 79 Appendix H Port Requirements Table 47. ESX/ESXi Port Requirements Description Ports Protocol Direction vSphere Client to ESX/ESXi host 443, 902, 903 TCP Incoming VM Console to ESX/ESXi host 903 TCP Incoming ESX/ESXi host and vCenter Heartbeat 902 UDP Incoming/ Outgoing ESX/ESXi host DNS client 53 UDP Outgoing ESX/ESXi host NTP client to NTP server 123 UDP Outgoing ESX/ESXi host NFS 111, 2049 TCP, UDP Outgoing vMotion between ESX/ESXi hosts 8000 TCP Incoming/ Outgoing HA between ESX/ESXi hosts 2050-2250, 8042-8045 TCP, UDP Incoming/ Outgoing ESX/ESXi host to Update Manager 80, 443, 9034 TCP Outgoing Update Manager to ESX/ESXi host 902, 9000-9010 TCP Incoming ESX/ESXi host CIM Client to Secure Server 5988, 5989 TCP Incoming ESX/ESXi host CIM service location protocol 427 TCP, UDP Incoming/ Outgoing
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 71 of 79 Table 48. vCenter Server Port Requirements Description Ports Protocol Direction vSphere Client to vCenter Server 443 TCP Incoming vSphere Web Access to vCenter Server 443 TCP Incoming VM Console to vCenter Server 902, 903 TCP Incoming ESX/ESXi host and vCenter Heartbeat 902 UDP Incoming/ Outgoing LDAP 389 TCP Incoming Linked Mode SSL 636 TCP Incoming ESX/ESXi 2.x/3.x host to legacy License Server 27000, 27010 TCP Incoming/ Outgoing Web Services HTTP 8080 TCP Incoming Web Services HTTPS 8443 TCP Incoming vCenter SNMP server polling 161 UDP Incoming vCenter SNMP client trap send 162 UDP Outgoing vCenter DNS client 53 UDP Outgoing vSphere Active Directory integration 88, 445 UDP, TCP Outgoing ODBC to MS SQL Server database 1433 TCP Outgoing Oracle Listener port to Oracle database 1521 TCP Outgoing
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 72 of 79 Table 49. vCenter Converter Standalone Port Requirements Description Ports Protocol Direction Converter Client (GUI) to Converter Server 443 (configurable) TCP Incoming Converter Server to remote Windows powered-on Machineremote agent deployment, Windows file sharing 445 and 139 TCP Incoming Converter Server to remote Windows powered-on Machineremote agent deployment, Windows file sharing 137 and 138 UDP Incoming Converter Server to remote Windows powered-on machineagent connection 9089 TCP Incoming Converter Server/Linux agent to remote Linux powered-on machine 22 TCP Incoming Converter Server/Agent to managed destinationVM creation/management (includes VM Helper creation/management) 443 TCP Incoming Windows powered-on machine to managed destinationhot cloneaccess (vCenter/ESX/ESXi) 443 TCP Incoming Windows powered-on machine to managed destinationhot clonecopy (ESX/ESXi) 902 TCP Incoming Windows powered-on machine to hosted destinationhot cloneWindows file sharing 445 and 139 TCP Incoming Windows powered-on machine to hosted destinationhot CloneWindows file sharing 137 and 138 UDP Incoming Helper VM to Linux powered-on machine hot clone 22 TCP Outgoing Converter Server/Agent to managed source/destinationVM importaccess (vCenter/ESX/ESXi) 443 TCP Incoming Converter Server/Agent to managed source/destinationVM importcopy from/to ESX/ESXi (Traffic from ESX/ESXi to ESX/ESXi direct for disk-based cloning only) 902 TCP Incoming VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 73 of 79 Converter Server/Agent to hosted source/destinationVM importWindows file sharing 445 and 139 TCP Incoming Converter Server/Agent to Hosted Source/DestinationVM ImportWindows file sharing 137 and 138 UDP Incoming
Table 50. vCenter Update Manager Port Requirements Description Ports Protocol Direction Update Manager to vCenter Server 80 TCP Incoming Update Manager to external sources (to acquire metadata regarding patch updates from VMware 80, 443 TCP Outgoing Update Manager client to Update Manager server 8084 TCP Incoming Listening ports for the Web server, providing access to the plug-in client installer and the patch depot 9084, 9087 TCP Incoming Update Manager to ESX/ESXi host (for pushing virtual machine and host updates/patches) 902 TCP Incoming
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 74 of 79 Appendix I Monitoring Configuration Table 51. SNMP Receiver Configuration SNMP Monitoring Server Port SNMP Community String emon01.domain.local 162 PRIV-RW
Table 52. vCenter SMTP Settings SMTP Server Sending Account Account to Receive Alerts mail03.domain.local Vsphere01.domain.local vsphereadmins@domain.local
Table 53. Physical to Virtual Windows Performance Monitor (Perfmon) Counters Old Physical Hardware Counter New Virtualization Aware Counter Processor - % Processor Time VM Processor - % Processor Time - Effective VM Speed in MHz (new) % Committed Bytes in Use Memory Active in MB - Memory Ballooned in MB (new) % Committed Bytes Memory Used in MB
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 75 of 79 Default vSphere Host Alarms to be Used Host hardware system board status Host power state Host memory status Host processor status Host disk status Host network status Host connection and power state Host memory usage Host CPU usage Host disk usage Host network usage Host storage status License error Cannot connect to network Cannot connect to storage
Default vSphere Cluster Alarms to be Used All HA hosts isolated Cluster deleted Cluster overcommitted HA admission control disabled HA agent unavailable HA disabled HA host failed HA host isolated Host resource overcommitted Insufficient failover resources No compatible host for secondary VM Virtual machine Fault Tolerance state changed Timed out starting secondary VM Cluster High Availability error Migration error
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 76 of 79 Default vSphere Datastore Alarms to be Used Datastore disk usage (%) Datastore state to all hosts Table 54. Modifications to Default Alarm Trigger Types Trigger Type Condition Warning Condition Length Alert Condition Length Host Memory Usage Is above 80 For 10 minutes 90 For 5 minutes Host CPU Usage Is above 80 For 10 minutes 90 For 5 minutes Datastore Disk Usage Is above 85 For 30 minutes 90 For 5 minutes
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 77 of 79 Appendix J Naming Conventions ESX/ESXi Host Naming Convention ESX hostnames follow this naming convention: ESX-<nn>-<domain>-local ESXi hostnames follow this naming convention: ESX-<nn>-<domain>-local where <nn> is a sequential number for the host <domain> is the Active Directory domain where the host resides vCenter Server Naming Convention VMware vCenter Servers follow this naming convention: vCenter-<location>-<nn>-<domain>.local where <location> is the datacenter location where the vCenter Server resides, using existing corporate naming conventions <nn> is a sequential number for the vCenter Server <domain> is the Active Directory domain where the vCenter Server resides Datacenter Naming Convention Datacenters follow this naming convention: <location>-<function> where: <location> is the datacenter location, using existing corporate naming conventions <function> is the role of the datacenter (for example, prod, dev, test, qa) Cluster Naming Convention Clusters follow this naming convention: <function>-<nn> where: <function> is the role of the department, such as Finance <nn> is a sequential number for the cluster
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 78 of 79 Virtual Switch Naming Convention Virtual switches follow this naming convention: <network-name>-<purpose>-<nn> where: <network-name> is the physical network associated with the vSwitch, for example, management, VM <purpose > is the type of network traffic (for example, production, test) <nn> is a sequential number for the vSwitch vSwitches must follow consistent, case-sensitive naming conventions to guarantee vMotion functionality along with other elements of the vSphere infrastructure.
VMware vSphere Plan and Design Services Architecture Design 2010 VMware, Inc. All rights reserved. Page 79 of 79 Appendix K Design Log DELETE THE FOLLOWING HIGHLIGHTED TEXT AFTER YOU READ IT This is sample of a design log for decisions made during your VMware vSphere engagement. The actual issues and resolutions for your customer will be specific to the issues that needed to addressed and solved. You will need to update this table. Table 55. Design Log ID Open Issue/Question Owner Close Date Answer/ Resolution Comments/ Proposed Value 101 102