What’s up with VADP backups and VDDK on vSphere 5.1?

VMware vSphere 5.1 has been in the market for more than a few months now and the interest in the new capabilities is high. Because of this the market saw many backup vendors rush to announce support for vSphere 5.1 in their VADP (vStorage APIs for Data Protection) integration. Everything looked clean and shiny and new.

On November 21, Symantec made an interesting announcement1. In a nutshell, the statement was that support for vSphere 5.1 would be delayed in its NetBackup and Backup Exec products. It was because they discovered issues while testing the VADP 5.1 API for integration. The API in the current form may introduce risk in performing consistent backups and ensuring reliable restores. All vendors receive the same API, not all vendors perform the same level of testing.

In order to explain the intricacies, first we need to take a quick look at how a backup product is integrated with VMware vSphere. With each release of vSphere, VMware publishes a set of APIs known as VMware APIs for Data Protection or VADP. One of the key components of VADP is Virtual Disk Development kit aka VDDK. This is the component through which third party code receives authenticated access to vSphere Datastores and virtual machine disk files. VMware makes this component available to its technology partners. Partners (backup product vendors in this case) ship this along with their product that has calls to vStorage APIs.

With each version of vSphere, an equivalent version of VDDK is released. The VDDK is generally backward compatible to one or more earlier versions of vSphere. For example, VDDK 5.1 supports2 vSphere 5.1, 5.0 and 4.1. VDDK 5.0 supports3 vSphere 5.0, 4.1, 4.0 and VI 3.5. Since the updated VDDK is required to understand the modified data structures in a new version of vSphere, lower versions of VDDK are in general not supported for accessing a higher version of vSphere. For example, VMware historically and currently (as of today) does not support the use of VDDK 5.0 to access datastores in vSphere 5.1.  VMware documents supported versions of vSphere for each of its VDDK versions in release notes.

The key to remember is the statement in bold face above. VMware does not support any violated combinations because of the risks and uncertainties. The partners are expected to ship the correct version of VDDK when they announce the availability of support for a given vSphere release.

What Symantec announced and VMware confirmed4 is that VDDK 5.1 has issues and hence the support for vSphere 5.1 in its products will be delayed. This makes sense since VDDK 5.1 is the only version currently allowed to access vSphere 5.1. The face-saving reactions from other vendors to this announcement revealed some of the dirty games and ugly truths to come out in the area of VADP/VDDK integration.


  1. Vendors were claiming support for vSphere 5.1 but still shipping VDDK 5.0 with their products. This is currently not supported by VMware because of the uncertainties.  This may change but at the time vendors claiming support, they were taking risks that typically are not acceptable in field of data protection business.
  2. Vendors were mucking with API calls and silently killing hung processes. That may work for an isolated or random hang. But will not work when there are repeatable hang situations like those observed in VDDK 5.1. Plus, there are performance and reliability concerns in abruptly ending sessions with vSphere.
  3. Most vendors weren’t testing all the edge cases and never realized the problems in VDDK 5.1, thus prematurely announcing support for 5.1


If your backup vendor currently supports vSphere 5.1, be sure to ask what their situation is.

Sources and references:

1. Quality wins every time: vSphere 5.1 support update, Symantec official blog.

2. VDDK 5.1 Release Notes, VMware Support resources

3. VDDK 5.0 Release Notes, VMware Support resources

4. Third-party backup software using VDDK 5.1 may encounter backup/restore failures, VMware Support KB

vPower: brand new solution, really?

When I started exploring AIX nearly eight years ago, there were two things that fascinated me right of the bat. I was already a certified professional for Solaris that time. I had also managed Tru64 UNIX and HP-UX mainly for Oracle workloads. Those used to be the time of tuning shared memory, message queue and semaphore parameters. During my days working as a contractor for a large financial institution and later for VERITAS/Symantec NetBackup technical support; tuning the UNIX system kernel for IPCS parameters were more of a norm than exception. AIX intrigued me because it featured a dynamic kernel! It was really a big deal for the kind of job I used to do!

The second thing that looked unique in comparison with rest of the UNIX platforms was AIX’s mksysb. In AIX, you could send the entire rootvg (all the boot files, system files and additional data file systems you may want to include in the root volume group) to a backup tape. When you need to restore your system from bare metal, you simply boot from tape medium and run the installer; your system is back to the same point in time when you did the mksysb backup. Furthermore, if needed, you can also boot from tape and restore selected files with a little help from tape positioning commands.

I went on to get certified on AIX, not just because of those two bells and whistles, but VERITAS Storage Foundation was expanding to AIX and it was a good thing to add AIX certification when we integrated its snapshot capabilities in NetBackup.

The mksysb started to become a bit obsolete for two reasons.

  1. It is expensive to have a standalone tape drive with every pSeries system. Not just because of the need for a tape drive on each system, rather the increased operational expenditure for the system administrators to manually track tapes with mksysb images for each system and also maintain a time-series catalog of all images.
  2. Enterprise data protection solutions like NetBackup added Bare Metal Restore (BMR) support. NetBackup BMR feature makes it possible to recover any physical system (be it AIX, HP-UX, Solaris, Linux, Windows…) from bare metal just by running a single command on master server to tell NetBackup that a client needs to be rebuild from bare metal. You also have the option to specify whether you need to bring the client to the most recent point in time (suitable in case of hardware failures) or a point in time from the past (suitable in case of logical corruptions that had happened before the most recent backup). After that you simply reboot the client. The client boots from network and recovers itself. The process is 100% unattended once the reboot is initiated.

What about virtual machines? You can indeed use NetBackup BMR feature on virtual machines. It is supported. The availability of deeper integration with VMware vADP and Hyper-V VSS makes it possible to perform agent-less backups of virtual machines whereby you could restore the entire VM or individual objects. Hence you don’t need it for VMs hosted by those hypervisors. You can use NetBackup BMR for VMs on other hypervisors like Citrix XenServer, IBM PowerVM, Oracle VM Server etc.  With NetBackup BMR and NetBackup Intelligent Deduplication, you have a solution no matter how many kinds of hypervisors are powering your clouds.

Why this story? Recently, during the after-party of a PR event hosted by Intel; I had a conversation with an old friend. He works for an organization who happens to be a partner for Veeam. He mentioned about Veeam and Visioncore are having a patent battle on the ability to run a system directly from the backup image. Veeam calls this feature as vPower, VisionCore calls it FlashRestore. This technology is really the virtual machine version of what IBM offered for AIX pSeries systems. You boot and run the system directly from the backup image and recover the whole system or selected files. The value additions like the flexibility to keep it running while being able to live migrate it to production storage comes from VMware’s innovative Storage vMotion technology which isn’t really something Veeam or VisionCore can take credit for. Visioncore may not have much difficulty fighting this battle.

We had a good laugh when we pulled Veeam’s marketing pitch on U-AIR which is nothing but running the VM from backup and copy required application files back to production VM over the wire. He raised his iPad to show Veeam’s datasheet to the group.

“vPower also enables quick recovery of individual objects from any virtualized application, on any OS. It’s a brand-new solution to the age-old problem of what to do when users accidentally delete important emails or scripts incorrectly update records.”

Brand new solution for the age-old problem, really?