VMware Fault Tolerance

VMware Fault Tolerance (FT) is a new feature available with VMware vSphere 4 that allows a virtual machine to continue running even when the underlying ESX Host fails.

Here's how it works:

When VMware FT is enabled on a virtual machine (called the Primary VM), a copy of the Primary VM (called the Secondary VM) is automatically created on another host, chosen by VMware Distributed Resource Scheduler (DRS). If VMware DRS is not enabled, the target host is chosen from the list of available hosts. VMware FT then runs the Primary and Secondary VMs in lockstep with each other – essentially mirroring the execution state of the Primary VM to the Secondary VM. In the event of a hardware failure that causes the Primary VM to fail, the Secondary VM immediately picks up where the Primary VM left off, and continues to run without any loss of network connections, transactions, or data. 

This new feature works very well but it does have it's limitations and drawbacks.

Probably one of the biggest gotchas for FT is the limited amount of processors that are supported.

ESX host processors must be VMware FT capable and belong to the same processor model family. VMware FT capable processors required changes in both the performance counter architecture and virtualization hardware assists of both AMD and Intel. These changes could only be included in recent processors from both vendors: third-generation AMD Opteron™ based on the AMD Barcelona, Budapest and Shanghai processor families; and Intel® Xeon® processors based on the Penryn and Nehalem micro-architectures and their successors.

For VMware FT to be supported, the servers that host the virtual machines must each use a supported processor from the same category as documented below:


Intel Xeon based on 45nm Core 2 Microarchitecture Category:

3100 Series

3300 Series

5200 Series (DP)

5400 Series

7400 Series

Intel Xeon based on Core i7 Microarchitecture Category:



3400 Series (Lynnfield)*

3500 Series

5500 Series

AMD 3rd Generation Opteron Category:

1300 and 1400 Series

2300 and 2400 Series (DP)

8300 and 8400 Series (MP)

* requires VMware vSphere 4.0 Update 1 or greater



For more details on CPU requirements please refer to http://kb.vmware.com/kb/1008027.

There are many other things to consider before implementing FT. I found this article while troubleshooting FT. It does a good job going over requirements and limitations:

VMware® Fault Tolerance Recommendations and Considerations on VMware vSphere™ 4

Great VMware vSphere Video for SMBs

...I stumbled across this video. It would probably be a great sales tool or primer for folks that aren't educated in VMware / vSphere. - Enjoy!

VMware vSphere SMB Video

VMware PSOD aka Purple Screen of Death

...I ran into a pretty good article on VMware.com that goes over the dreaded PSOD. It explains how do decode that wonderful screen of eclectic info.


VMware KB Article 1005184 PSOD

VirtualCenter Server Service fails to start

 After upgrading to vCenter Server 4.0 Update 1, the VirtualCenter Server Service failed to start.  

The VirtualCenter log files contain entries similar to:


[2009-11-28 14:03:32.282 05756 info 'ProxySvc'] vmacore/ssl/useSSLCtxPool: true
[2009-11-28 14:03:32.282 05756 info 'ProxySvc'] vmacore/ssl/serializeServerHandshake: false
[2009-11-28 14:03:32.282 05756 info 'ProxySvc'] Entry matching VC_SERVER_NAME:8089/ already exists.

The fix can be found in KB Article: 1016116.
 VMware KB Article 1016116

Restarting the Management agents on ESX

Log in to your ESX Server as root from either an SSH session or directly from the console of the server.
Type service mgmt-vmware restart
Caution: Ensure Automatic Startup/Shutdown of virtual machines is disabled before running this command or you risk rebooting the virtual machines.
Press Enter
Type service vmware-vpxa restart.
Press Enter.
Type logout and press Enter to disconnect from the ESX host.

Note: Information found at http://www.vmware.com/

Applying patches and hotfixes to vCenter Server when using vCenter Server Heartbeat.

I was given the task of applying the SP1 patch on our vCenter Server Heartbeat Cluster (VSH). VSH is a great product but does add some complexity when it comes to applying patches.


Here are the required steps to follow in order to update a vCenter Server Heartbeat Cluster with minimal interruption:

1. Download and install the patches or hotfixes, but do not reboot the server if instructed to do so.
2. Launch the vCenter Server Heartbeat Console.
3. Perform a switchover using the vCenter Server Heartbeat Console to make the Secondary server active.
4. Note: At this point users may notice a slight interruption to the protected application.
5. When switchover is complete, shutdown vCenter Server Heartbeat. Click Shutdown vCenter Server Heartbeat, but leave protected applications running.
6. Reboot the Primary server, if prompted at step 1, to complete the installation of the patches or hotfixes.
7. Verify all patches and hotfixes are installed properly on the Primary Server..
8. Start vCenter Server Heartbeat on the Secondary server and also on the Primary server, if no reboot was required at step 6, allowing the servers to synchronize.
9. Allow servers to connect and synchronize with the Secondary as active and the Primary as passive.
10. When synchronization is complete, launch the Update on the Secondary (active) server.
11. Download and install the patches or hotfixes, but do not reboot the server if instructed to do so.
12. Launch the vCenter Server Heartbeat Console.
13. Perform a switchover using the vCenter Server Heartbeat Console to make the Primary server active again.
14. Note: At this point users may notice a slight interruption to the protected application.
15. When switchover is complete, shutdown vCenter Server Heartbeat. Click Shutdown vCenter Server Heartbeat, but leave protected applications running.
16. Reboot the Secondary server, if prompted at step 11, to complete the install of the patches or hotfixes.
17. Verify all patches are installed properly.
18. Start vCenter Server Heartbeat on the Secondary server and also on the Primary server, if no reboot was required at step 16, allowing the servers to synchronize.
19. Allow servers to connect and synchronize with the Primary as active and the Secondary as passive.

Go get a cup of coffee and prepare for your next daily challenge.

This information was found on vmware.com.

Aligning New Virtual Machine File Systems

Here are the steps that need to be followed to align a New Virtual Machine File System:

1. Log into the virtual machine using admin credentials.
2. Open a coomand prompt and type diskpart.
3. Type list disk and press Enter.
4. Type select disk 1 and press Enter.
5. Type create partition primary align=64, and press Enter.
6. Type assign letter = X, where X is an open drive letter that can be assigned.
7. Type list part to verify the 64kb offset for the new partition.
8. Format the partition.

Site Recovery Manager 4.0 Performance and Best Practices

SRM is one of  VMware's products that really doesn't get it's fair share of respect. Just the idea of a product that can migrate up to 1000 VMs to a remote site in an automated fashion is incredible. Here is the latest Best Practices Doc from VMware.

SRM Best Practices

ESX 3.5 Update 5 Available for Download

The long awaited ESX 3.5 Update 5 is now Available. Supporting some new features, processors and even a few new operating systems.


Notes From VMware:


Not all combinations of VirtualCenter and ESX Server versions are supported. All of these highlighted features are available only if you are using VirtualCenter 2.5 Update 5 with ESX Server 3.5 Update 5. See the ESX Server, VirtualCenter, and VMware Infrastructure Client Compatibility Matrixes for more information on compatibility.
VMware recommends VMware Tools upgrade for this version of ESX Server.


The following information provides highlights of some of the enhancements available in this release of VMware ESX Server:

VMCI Socket Performance

VMware Virtual Machine Communication Interface

Great Article on VMCI. Here is the description:

The VMCI (Virtual Machine Communication Interface) device allows fast, efficient communication between virtual machines running on the same host, without using the guest networking stack. This paper presents VM-VM performance results using VMCI Sockets and compares these results to the VM-VM performance achieved using regular TCP/IP sockets.

Dynamic Storage Provisioning

Dynamic Storage Provisioning

A Great Intro into thin Provisioning in vSphere. Here's the description:

This paper examines the capabilities of thin provisioning, enhancements to VMware vCenter™ in vSphere, and addresses best practices to ensure thin provisioned virtual disks are a benefit and work to their full potential. This paper also explains how thin virtual disks fit into the overall storage stack, and how virtual disk thin provisioning operates with array-based thin provisioning and dynamic expansion of virtual disks, VMFS volumes, and LUNs within the array.

Beau's Weekly Scripting Challenge - 12/07/2009

Question: How do you delete stale vswp files?

Answered by: Beau Newcomb

find . -name *.vswp \( ! -regex '.*/\..*' \)

The regex piece just ignores any folders with a . (aka, .snapshot). That would leave us with a list of vswaps:

./TESTVMW0018/TESTVMW0018-94a76642.vswp
./TESTVMW0074/TESTVMW0074-94a76704.vswp
./TESTVMW0077/TESTVMW0077-94a76707.vswp

Those VMs were vmotioned, and then I ran the same command, but piped it to a delete to clean them up:

find . -name *.vswp \( ! -regex '.*/\..*' \) while read vswp; do rm -rf $vswp; done

I then ran the original command again and turned up no results.

BEAU Thanks for spending the time to do this. -Scott