The Big Hole in EMC Big Data backup story

It is one of the crucial roles for the marketing team in any organization to communicate the value of its products and services. It is not uncommon (pardon the double negative) for organizations to show the best side of its story while deliberately hiding the weaker aspects through fine prints. The left side of the picture below is the snapshot of breakfast cereal (General Mills’ Total) that came with my breakfast order in Sheraton while travelling on business.

EMC appears to have a Big Hole in its Big Data Backup

EMC appears to have a Big Hole in its Big Data Backup

Note that General Mills had claimed 100% of daily value of 11 vitamins and minerals but with an asterisk. The claim is true only if I consume 53g serving, but the box has only 33g!

Although I may have felt a bit taken back as a consumer, I enjoyed giving a bit of hard time to my General Mills friends and I moved on. This is a small transaction.

What if you were responsible for a transaction worth tens of thousands of dollars and were pitched a glass half-full story like this? It does happen. That General Mills cereal box is what came to my mind when I saw this blog from EMC on protecting Big Data (Teradata) workloads using EMC ‘Big Data backup solution’.

General Mills had the courtesy put the fine print that part of the vitamins and minerals are missing from its box. EMC’s blog didn’t really call out what was missing from its ‘box’ aka Data Domain device to protect Teradata workload using Teradata Data Stream Architecture. In fact it is missing the real brain of the solution: NetBackup!

First a little bit of history and some naked truth. Teradata had been working with NetBackup for over a decade to provide data protection for its workloads. In fact, Teradata sells the NetBackup Agent for Teradata for its customers. This agent pushes the data stream to NetBackup media servers. This is where the real workload aware intelligence (the real brain for this Big Data backup) is built. Once NetBackup media server receives the data stream it can store it on any supported storage: NetBackup Deduplication Pool, NetBackup Advanced Disk Pool, NetBackup OpenStorage Pool or even on a tape storage unit! When it comes to NetBackup OpenStorage Pool, it does not matter who the OpenStorage partner is; it can be EMC Data Domain, Quantum DXi,… The naked truth is that the backend devices are dumb storage devices from the view of NetBackup Agent for Teradata (the Teradata BAR component depicted in the blog).

EMC’s blog appears to have been designed to mislead the reader. It tends to imply that there is some sort of special sauce built natively into Data Domain (or Data Domain Boost) for Teradata BAR stream. The blog is trying to attach EMC to Big Data type workloads through marketing. May I say that the hole is quite big in EMC’s Big Data backup story!

I am speculating that EMC had been telling this story for a while in private engagements with clients. Note that the blog is simply displaying some of EMC’s slides that are marked ‘confidential’. The author forgot to remove it before publishing it. In closed meetings with joint customers of Teradata and NetBackup, a slide like this will create the illusion that Data Domain has something special for Teradata backup. Now the truth just leaked!

NetBackup Accelerator vs. Simpana DASH Full

I want to start this blog with a note.

I mean no disrespect to CommVault as a company or its engineers innovating its products. Being an engineer myself by trade, I do understand that innovations are triggered by market demands and there is always room for improvements in any product. This blog is entirely my own opinions.

As most of you guys reading this blog know, I also write for official Symantec blogs. I recently got an opportunity to take readers of Symantec Connect on a deep dive into one of the major features in NetBackup 7.6 for VMware vSphere and vCloud environments. It is primarily targeted for users of NetBackup who knows its nuts and bolts. A couple of employees from a CommVault read the blog. It is natural in competitive intelligence world to look for weak spots or things that can be selectively pointed out to show parity. It is part of their job and I respect it. However it appeared that they wanted to claim parity for Simpana with NetBackup Accelerator for VMware based on two statements (tweets, to be precise!). While asking to elaborate, the discussion went on a rat hole with statements made out of context and downright unprofessional. Hence here I go with an attempt to compare Simpana 10 with NetBackup 7.6 on the very topic discussed in official blog.

Claims to equate parity with NetBackup Accelerator for VMware

  1. (Not explicitly stated) Simpana supports CBT
  2. Simpana had ‘block detection’ for over a year
  3. Simpana does synthetics

The attempt here is to check all the boxes to claim parity while at times people do miss the big picture! At times they were equating apples to oranges. Hence I am going to attempt to clarify this as much as possible using Simpana language for the benefit those two employees.

Simpana supports CBT: Of course, every major vendor supports it. It is an innovation from VMware. The willingness to support a feature from vStorage APIs is important to protect VMware virtual machines.

What sets NetBackup 7.6 apart from Simpana 10 in this case is that Simpana’s implementation of CBT is limited to recovering an entire VM or individual files from the VM. If you have enterprise applications (e.g. Microsoft Exchange, Microsoft SQL Server etc.), you must stream data through an agent inside the guest to protect those applications and perform granular recovery. The value of CBT is to minimize data processing and movement load on production VMs while performing backups. A virtual machine’s operating system binaries and related files are typically static and CBT won’t add much value there. The real value comes from daily changes to disk blocks by applications! That means ZERO value in Simpana to protect enterprise applications with its implementation of vSphere CBT.

Simpana had block detection for over a year,  Simpana does synthetics: The employee is trying to add a check box for Simpana next to NetBackup’s capability to make use of Symantec V-Ray to detect deleted blocks. Nice try!

First and foremost, the block optimization technique described in my blog is present in NetBackup since 2007, with version 6.5.1 when Symantec announced support for VMware Virtual Infrastructure 3. Congratulations on trying to claim that Simpana had this capability after half a decade! But wait…. We are talking about apple and orange here.

This technique had been available for both full and incremental backup schedules. It works no matter where backups are going to, disk, deduplicated disk, tape or cloud. NetBackup’s block optimization happens closer to the data source. Thus it detects deleted blocks at the backup host so that the deleted blocks never appear in SAN or LAN traffic to the backup storage. That is optimization for processing-power, interconnect-bandwidth and storage!

CommVault employee was in a hurry to equate this to something Simpana caught up recently.  This is what I believe he is referring to. (I am asking him to tweet back if there is anything else).  Quoted from Simpana 10 online documentation.

DASH Full is a read optimized Synthetic Full operation which does not require traditional full backups to be performed. Once the first full backup is completed, changed blocks are protected during incremental or differential backups. A DASH Full will run in place of traditional full or synthetic full. This operation does not require movement of data. It will simply update indexing information and the deduplication database signifying that a full backup has been completed. This will significantly reduce the time it takes to perform full backups.

There are so many things I want to say about this, but I am trying to be concise here with bullet points.

  • What Simpana has here is an equivalent of NetBackup OpenStorage Optimized Synthetics that was introduced in NetBackup 6.5.4 (in 2009). While NetBackup still supports this capability, Symantec had taken this to the next level with NetBackup Accelerator. For the record, NetBackup Accelerator is also backed by Optimized Synthetics and hence the so-called ‘block detection’ is there in NetBackup since 2009.
  • The optimization I was talking about was the capability to detect deleted blocks from the CBT data stream while CommVault is touting about data movement within backup storage!
  • The DASH full requires incremental backups and separate schedules for synthetic backups. NetBackup Accelerator eliminates this operational inefficiency by synthesizing full image inline using the resources needed for an incremental backup.
  • If you are curious about how NetBackup Accelerator in general is different from Optimized Synthetics (or DASH Full), this blog would help.
  • Last but not the least, did I say that NetBackup Accelerator for VMware works with enterprise applications as well? Thus both CBT and deleted blocks detection (both relevant to applications that does the real work inside VM) adds real value for NetBackup Accelerator

Software Defined Storage: Next Big Thing? Or is it already here?

Software Defined Storage

Software Defined Storage

  Walk into a technology trade show with a bottle of tequila and a shot glass. Take a shot each time you hear the phrase ‘software defined’, you would need a cab to get back home. The new buzzword is ‘software defined’ and storage vendors are making ways to fit it in. It doesn’t matter whether it is one of those established players or upstarts. If it isn’t software defined, it is not cool.

What is SDS? There are many vendor specific definitions and interpretations for SDS. Industry analysts like Gartner and IDC have their own versions too. Hence I would just say what the attributes and values are expected in general from SDS.

Abstraction: Data pane and control pane are separated. In other words, the storage management is decoupled from the actual storage itself.

Backend heterogeneity:  Storage is served by any kind of storage from any vendor including commodity storage.

Frontend heterogeneity: Storage is served to any kind of consumers (operating systems, hypervisors, file services etc.)

SDS is a broker: This was the statement from Gartner. SDS brokers storage services, independent of where data is placed and how it is stored, through software that in turn will translate those capabilities into storage services that meet a defined policy or SLA.

Logical volume managers in operating systems had provided the attributes 1 and 2 since 80s. The cross-platform volume manager from Veritas (acquired by Symantec) brought 3 into the mix with the introduction of portable data containers (also known as cross-platform data sharing disks) in 2004. Other notable examples fitting the requirements 1,2 and 3 are IBM’s SAN Volume Controller (SVC) offering and NetApp’s V-series.

Now let us take a look at the final attribute to fit something in SDS, the broker role. Surprisingly, Veritas Volume Manager (VxVM) meets that requirement as well. Storage services like file service, deduplication, storage-tiering etc. are provided by Veritas Volume Manager independent of where and how it is stored.

Let me elaborate this further to prove this point. VxVM abstracts storage from any array or commodity storage. This is old news. What is new about the latest version of VxVM (part of Symantec Storage Foundation 6.1 powered by Veritas) is that it can now provide a single name space (data service) across multiple nodes without the requirement to have disks shared via a storage area network. This completes the final requirement to fit the attributes required by SDS.

The point of the blog is not to underestimate the significance of SDS, rather to conclude that SDS may already be here depending on your definition and interpretation! Let us think beyond vendor presentations. How is EMC’s ViPR different from what NetApp V-Series had been offering for years? Should Flash be on hybrid array (e.g. Nimble Storage), all flash (e.g. Pure Storage) or close to system (e.g. Nutanix)? Or should you adopt something that gives the flexibility to choose (Symantec Storage Foundation 6.1, pardon my pitch). It truly depends on your business requirements, but I would say that Flash Anywhere might provide flexibility while storage industry is looking for the winner.

Data Domain: The TiVo in data deduplication market

It is that time of the year when Christmas shopping is in full swing. If you rewind time by just six to eight years, TiVo’ing was a verb used widely and my team was not an exception. TiVo was a must have gadget in the house. My manager once said, “it is not easy to find a technical gadget that my wife would love, but TiVo was something she could operate and enjoy without being coached.”

TiVo was the gold standard of digital video recorders. Consumers were willing to pay a premium for the box upfront and sign up for monthly fees to get ‘Tivo Service”, the data service using which TiVo populated programming schedules and tasks. These costs are in addition what consumers may be paying for Cable or Satellite services.

Cable and satellite operators came with built-in DVRs in their set top boxes. Still, TiVo used to be the star. It had 4.36 million subscribers in 2006. The technology and usability for DVRs from competitors was so poor that consumers continued to pay a premium to enjoy hassle free experience in TiVo.

Fast forward a few years from its peak in 2006, TiVo stumbled into identity crisis. The competition came not just from Cable and Satellite providers who matched the simplicity of TiVo in their all-in-one set top offerings, but it also came indirectly from streaming services (NetFlix, Amazon, Hulu), cheaper purpose built set top boxes for streaming (Apple TV, Roku) and multi-purpose devices (Wii, PlayStation 3, Xbox). Now TiVo is struggling to stay relevant. It is no more asking for upfront premium for the box, if you commit to a 2-year subscription the box is yours. It partners with competitors to bring their services into TiVo. If there was a market for ‘Digital Video Recorders’, now it is squeezed by players from adjacent markets.

Holiday Offer from, seen on November 27, 2012

Holiday Offer from, seen on November 27, 2013


Today Deduplication Storage is such a market where Data Domain used to be the TiVo. It was a powerful but yet simple device that an IT administrator could manage without reading its manual. It moved out tape as main backup storage medium once it started to integrate with market leading backup applications, especially with Symantec’s NetBackup through OpenStorage. EMC had to pay a fortune (and fight with NetApp) to acquire this technology, but it paid off, as Data Domain was the only cash cow in EMC’s Backup and Recovery Services division.

Data Domain could ask for a premium as other players in the market couldn’t match the technology and simplicity. But now… the tide is changing…

Direct competitors are getting their act together. HP matched Data Domain’s scale and performance and added high availability on top of it. Symantec launched integrated all-in-one appliances with content aware deduplication built-in. Most backup software vendors have deduplication available as a feature. Even standard file systems (Symantec’s VxFS, Microsoft’s NTFS, Oracle’s ZFS) are now including deduplication. Now “data deduplication as a market” is being squeezed by competitors and adjacent players.  Customers are less and less likely pay a premium for deduplication, as it is becoming a commodity.

TiVo managed to kill VHS tapes, which was the primary recording device for television shows. EMC touted Data Domain as a tape killer in backup industry. While disk based solutions have indeed limited the value of tape (now it is used primarily for long term retention), Data Domain as a standalone premium deduplication storage device may be extinct even before tape gives up its last breath. Time will tell.





High Availability for Business Critical Applications on VMware vSphere

In the last blog we talked about VMware vSphere HA and FT. As we discussed, vSphere HA is quite impressive in protecting against infrastructure failures at a reasonable cost. vSphere FT, on the other hand, has very limited use cases. However, none of these solutions are sufficient to meet high availability requirements for business critical applications with demanding service level agreements.

  1. Neither vSphere HA nor FT has application awareness. These technologies monitor just the container (the virtual machine). If an application or a resource that it depends on goes down, these technologies cannot detect and remediate the issue.
  2. Both technologies cannot provide availability during planned downtimes. If the application or operating system needs to be patched, the application will not be available to users.
  3. The remediation in vSphere HA requires restarting guest operating system that can be time consuming. This poor RTO may not be suited for enterprise applications.

This is where Symantec comes to rescue VMware vSphere administrators. Symantec has two products to fill these gaps so that organizations can confidently virtualize business critical applications. Thus you get to enjoy the agility and cost efficiency of VMware vSphere without compromising enterprise availability.

 Symantec ApplicationHA: This solution solves problem 1 given above. Symantec ApplicationHA monitors designated application and resources (e.g. disk, volume, file system, network…). If a failure is detected Symantec ApplicationHA can restart the application and its resources in a pre-defined order. The application monitoring is quite efficient and foolproof. For example, if you are monitoring MS SQL Server, you can configure the ApplicationHA agent to login and logout from the database as if it were a regular database user. If the application restart fails (it can attempt application restarts for a configured number of times), Symantec ApplicationHA will send a trigger to vSphere HA (if available) to restart the VM on the same or on a different host.

Symantec ApplicationHA has support for over 21 business critical applications. Moreover, it provides a framework to create custom agents for homegrown applications as well. Symantec has been in the business of HA and DR for long time with superb reputation for HA agents.

Symantec Cluster Server, powered by Veritas: This product solves all the three problems stated earlier. This can work with or without vSphere HA. The application monitoring and remediation workflow is similar to that of Symantec ApplicationHA. In fact, the agents for both the products are the same. What is different about Symantec Cluster Server is its ability to migrate just the application and its resources to a standby VM as part of remediation.  There is no need to wait for VM to restart thereby significantly reducing the downtime and improving RTO.

The ability to migrate application to another VM also mitigates the downtime normally incurred for planned activities like applying maintenance updates. You can update patches on standby node and migrate the application. Considering the long patching processes for modern operating systems, you definitely don’t want to deploy business critical applications on vSphere without the availability from Symantec Cluster Server.

Symantec Cluster Server for VMware is purpose built for virtual environments. When compared to traditional clusters like Windows Failover Cluster (formerly known as Microsoft Cluster Server), Symantec Cluster Server gives you high availability without compromising the perks of virtualization. For example, Windows Failover Cluster requires you to create VMs with physical RDM (raw device mapping) disks. If you use RDM, the flagship vSphere capabilities like vMotion, DRS, vStorage API based backups etc. are lost. Symantec Cluster Server has hot-plug APIs to work in VMFS and NFS based datastores.

 Note: VMware has released product called vSphere App HA with vSphere 5.5 release. Lorenzo has written a great blog where Symantec Application HA and vSphere App HA are compared in detail. Check it out here.

Run baby run! High Availability for business critical applications in virtualized environments

Most of you are on a journey to a software defined data center. Some of you used virtualization to consolidate infrastructure to reduce capital expanses. Some of you may be virtualizing (or starting to think virtualizing) business applications to take advantage of the agility and flexibility that virtualization brings. Naturally, one thing you may be worried a lot is system and application availability if you have reached that part of the journey.

The good news is that VMware is not a stranger to HA. VMware vSphere includes a feature named vSphere HA (formerly VMware HA) that protects VMs against hardware failures.  Two or more ESXi hosts can form an HA cluster. vSphere HA provides the following values.

  1. Decent protection against hardware failures (ESXi host failures). When a host fails, the virtual machines on that host can be restarted on another ESXi host sharing the same data store.
  2. Limited protection against guest OS failures. The VMware tools running on guess operating system sends heartbeats to vSphere HA. If heartbeat stops (e.g. the guest operating system is hung), vSphere HA can restart the VM on the same or on a different ESXi host.

The use of vSphere HA depends on the service level agreement (SLA) between IT department and business unit. In most development/test workloads, vSphere HA is good enough as the services can be resumed in less than 10 minutes. The main bottleneck here is the time it takes to reboot the guest operating system.

Another solution is vSphere Fault Tolerance (vSphere FT).  It creates and maintains an additional copy of the VM being protected. It provides continuous availability by ensuring that the states of the primary and secondary VMs are identical at any point in the instruction execution of the virtual machine. However, vSphere FT is not for everyone. Although its protection against hardware failures is impeccable, its protection against OS and applications misbehavior is extremely limited. The cost of operating two virtual machines (and related storage) and other limitations like lack of support for vStorage APIs makes vSphere FT suitable for very limited use cases.

Both vSphere HA and vSphere FT lacks something quite important when it comes to protecting business critical workloads, viz. application awareness. Let us say that you are running an instance of Oracle with a few databases inside a virtual machine. What happens if an Oracle instance fails? What happens of an instance loses access to underlying storage? Neither VMware HA nor FT detects it and hence downtime will be incurred. Downtime = Lost revenue.

There is another weakness in vSphere HA and vSphere FT solutions. It does not protect applications against planned downtimes. When you need to patch, upgrade or perform any other maintenance task related to components within the guest (operating system binaries, application binaries etc.) you must shutdown the application that may be costly for tier 1 business critical applications.

ScenariovSphere HAvSphere FT
Detect host failureVMs are restarted on another host (Recovery time = restart time)The VM executing instructions in lockstep on surviving host takes over (Recovery time is near zero)
Detect VM failure (VM not sending heart beats, OS hung) VM is restartedNo protection likely as both VMs are in lockstep
Detect Application FailureNo ProtectionNo Protection
Compatibility with vMotionYesYes
Compatibility with vStorage APIs for Data Protection (VADP)YesNo (in guest backup agent required)
Avoiding Planned Downtime (patching, upgrades etc.) Planned downtime cannot be avoidedPlanned downtime cannot be avoided

Symantec has solutions to tackle these types of scenarios. One was jointly developed with VMware. The second one comes from a time-tested solution that was ported to support vSphere platform. Let us look at each of them in another blog.

What’s up with VADP backups and VDDK on vSphere 5.1?

VMware vSphere 5.1 has been in the market for more than a few months now and the interest in the new capabilities is high. Because of this the market saw many backup vendors rush to announce support for vSphere 5.1 in their VADP (vStorage APIs for Data Protection) integration. Everything looked clean and shiny and new.

On November 21, Symantec made an interesting announcement1. In a nutshell, the statement was that support for vSphere 5.1 would be delayed in its NetBackup and Backup Exec products. It was because they discovered issues while testing the VADP 5.1 API for integration. The API in the current form may introduce risk in performing consistent backups and ensuring reliable restores. All vendors receive the same API, not all vendors perform the same level of testing.

In order to explain the intricacies, first we need to take a quick look at how a backup product is integrated with VMware vSphere. With each release of vSphere, VMware publishes a set of APIs known as VMware APIs for Data Protection or VADP. One of the key components of VADP is Virtual Disk Development kit aka VDDK. This is the component through which third party code receives authenticated access to vSphere Datastores and virtual machine disk files. VMware makes this component available to its technology partners. Partners (backup product vendors in this case) ship this along with their product that has calls to vStorage APIs.

With each version of vSphere, an equivalent version of VDDK is released. The VDDK is generally backward compatible to one or more earlier versions of vSphere. For example, VDDK 5.1 supports2 vSphere 5.1, 5.0 and 4.1. VDDK 5.0 supports3 vSphere 5.0, 4.1, 4.0 and VI 3.5. Since the updated VDDK is required to understand the modified data structures in a new version of vSphere, lower versions of VDDK are in general not supported for accessing a higher version of vSphere. For example, VMware historically and currently (as of today) does not support the use of VDDK 5.0 to access datastores in vSphere 5.1.  VMware documents supported versions of vSphere for each of its VDDK versions in release notes.

The key to remember is the statement in bold face above. VMware does not support any violated combinations because of the risks and uncertainties. The partners are expected to ship the correct version of VDDK when they announce the availability of support for a given vSphere release.

What Symantec announced and VMware confirmed4 is that VDDK 5.1 has issues and hence the support for vSphere 5.1 in its products will be delayed. This makes sense since VDDK 5.1 is the only version currently allowed to access vSphere 5.1. The face-saving reactions from other vendors to this announcement revealed some of the dirty games and ugly truths to come out in the area of VADP/VDDK integration.


  1. Vendors were claiming support for vSphere 5.1 but still shipping VDDK 5.0 with their products. This is currently not supported by VMware because of the uncertainties.  This may change but at the time vendors claiming support, they were taking risks that typically are not acceptable in field of data protection business.
  2. Vendors were mucking with API calls and silently killing hung processes. That may work for an isolated or random hang. But will not work when there are repeatable hang situations like those observed in VDDK 5.1. Plus, there are performance and reliability concerns in abruptly ending sessions with vSphere.
  3. Most vendors weren’t testing all the edge cases and never realized the problems in VDDK 5.1, thus prematurely announcing support for 5.1


If your backup vendor currently supports vSphere 5.1, be sure to ask what their situation is.

Sources and references:

1. Quality wins every time: vSphere 5.1 support update, Symantec official blog.

2. VDDK 5.1 Release Notes, VMware Support resources

3. VDDK 5.0 Release Notes, VMware Support resources

4. Third-party backup software using VDDK 5.1 may encounter backup/restore failures, VMware Support KB

Dear EMC Avamar, please stop leeching from enterprise vSphere environments

VMware introduced vStorage APIs for Data Protection (VADP) so that backup products can do centralized, efficient, off-host LAN free backup of vSphere virtual machines.

In the physical world, most systems have plenty of resources, often underutilized. Running backup agent in such a system wasn’t a primary concern for most workloads. The era of virtualization changed things drastically. Server consolidation via virtualization allowed organizations to get the most out of their hardware investment. That means backup agents do not have the luxury to simply take up resources from production workloads anymore as the underlying ESXi infrastructure is optimized and right-sized to get line of business applications running smoothly.

VMware solved the backup agent problem from the early days of ESX/ESXi hosts. The SAN transport method for virtual machine backup was born during the old VCB (VMware Consolidated Backup) days and further enhanced in VADP (vStorage APIs for Data Protection). The idea is simple. Let the snapshots of virtual machine be presented to a workhorse backup host and allow that system do the heavy lifting of processing and moving data to backup storage. The CPU, memory and I/O resources on ESX/ESXi hosts are not used during backups. Thus the production virtual machines are not starved for hypervisor resources during backups.

For non-SAN environments like NFS based datastores, the same dedicated host can use Network Block Device (NBD) transport to stream data through management network. Although it is not as efficient as SAN transport, it still offloaded most of the backup processing to the dedicated physical host.

Dedicating one or more workhorse backup systems to do backups was not practical for small business environments and remote offices. To accommodate that business need, VMware allowed virtual machines to act as backup proxy hosts for smaller deployments. This is how hotadd transport was introduced.

Thus your backup strategy is to use a dedicated physical workhorse backup system to offload all or part of backup processing using SAN or NBD transports. For really small environments, a virtual machine with NBD or hotadd transport would suffice.

Somehow EMC missed this memo. Ironically, EMC had been the proponent of running Avamar agent inside the guest instead of adopting VMware’s VADP. The argument was that the source side deduplication at Avamar agent minimizes the amount of data to be moved across the wire. While that is indeed true, EMC conveniently forgot to mention that CPU intensive deduplication within the backup agent would indeed leech ESXi resources away from production workloads!

Then EMC conceded and announced VADP support. But the saga continues. What EMC had provided is hotadd support for VADP. That means you allocate multiple proxy virtual machines even in the case of enterprise vSphere environments. Some of the best practice documents for Avamar suggest deploying a backup proxy host for every 20 virtual machines. Typical vSphere environment in an enterprise would have 1000 to 3000 virtual machines. That translates to 50 to 150 proxy hosts! These systems are literally the leach worms in vSphere environment draining resources that belong to production applications.

The giant tower of energy consuming nodes in Avamar grid is not even lifting a finger in processing backups! It is merely a storage system. The real workhorses are ESXi hosts giving in CPU, memory and I/O resources to Avamar proxy hosts to generate and deduplicate backup stream.

The story does not change even if you replace Avamar Datastore with a Data Domain device. In that case, the DD Boost agent running on Avamar proxy hosts are draining resources from ESXi to reduce data at source and send deduplicated data to Data Domain system.

EMC BRS should seriously look at the way Avamar proxy hosts with or without DD Boost are leaching resources from precious production workloads. The method used by Avamar is recommended only for SMB and remote office environments. Take the hint from VMware engineering as to why Avamar technology was borrowed to provide a solution for SMB customers in VMware Data Protection (VDP) product. You can’t chop a tree with a penknife!

The best example for effectively using VADP for enterprise vSphere is NetBackup 5220. EMC BRS could learn a lesson or two from how Symantec integrates with VMware in a much better way. This appliance is a complete backup system with intelligent deduplication and VADP support built right in for VMware backups.  This appliance does the heavy lifting so that production workloads are unaffected by backups.

How about recovery? For thick provisioned disks SAN transport is indeed the fastest. For thin provisioned disks, NBD performs much better. The good news on Symantec NetBackup 5220 is that the user could control the transport method for restores as well. You might have done the backup using SAN transport, however you can do the restore using NBD if you are restoring thin provisioned virtual machines. For Avamar, hot-add is the end-all for all approaches. NBD on a virtual proxy isn’t useful, hence using that is a moot point when the product offers just virtual machine proxy for VADP.

The question is…

Dear EMC Avamar, when will you offer an enterprise grade VADP based backup for your customers? They deserve enterprise grade protection for the investment they had done for large Avamar  Datastores and Data Domain devices.



VMware announces vSphere Data Protection (VDP), what is in it for you?

vSphere Data Protection (VDP) is VMware’s new virtual backup appliance for SMB available in VMware vSphere 5.1. It replaces the older VMware Data Recovery (vDR) product. There had been a number of confusions around this announcement; partly due the way EMC, VMware’s parent company, made some press releases.

Is VDP the same as EMC’s Avamar Virtual Edition (Avamar VE)?

No, it is not. VDP is a product from VMware. The only technology VMware had used from Avamar is its deduplication engine. The older vDR had limited dedupe capabilities as it was mainly coming from change block tracking (CBT) in vStorage APIs for Data Protection (VADP). With Avamar’s technology, VDP now provides variable block based deduplication.

I heard that I can upgrade from VDP to EMC Avamar if I need to grow beyond 2TB, is that true?

No, VDP is not a ‘lite’ version of Avamar. It is a different product altogether.

What are my options if I need to grow beyond 2TB?

You could add additional VDP appliances. Up to 10 VDP appliances are supported under one vCenter server. However, these are separate islands of storage. These appliances do not provide global deduplication among these storage pools.

Having said that it is more likely for you to hit other limitations in VDP before hitting the 2TB limit. Note that Avamar based deduplication engine is suitable only for SMBs who could afford to have black out windows and maintenance windows in their backup solution. These are the periods of time where the house keeping work is being done by dedupe engine.  The system is not available for running backup jobs.

Only 8 virtual machines can be backed up concurrently that might increase backup windows. There is no SAN transport capability to offload production ESXi hosts from backup tasks. There is no good way to make additional copies for redundancy or extended retention like replication to remote location or cloud. VMware has made it clear that VDP is truly for SMBs and encourages customers to look at enterprise class backup solutions from partners for larger environments.

Why would EMC let VMware use its Avamar technology at no additional cost to customers? Is EMC trying to promote its products?

Just like how Windows/UNIX/Linux operating environments provide basic utilities for backups, VMware had always provided basic backup solution with its offerings. In the days of ESX service console, the Linux based console provided tools like tar and cpio. With ESXi where service console is no more, vDR was brought to the table. vDR had its limitations. Now the choice is to innovate vDR or license a relatively mature technology. As parent company has a solution, VMware went the route of taking Avamar dedupe engine for storage and build its own capabilities for scheduling backups and managing recovery points.

EMC’s Avamar is a popular product in small environments. Although EMC had been trying hard to make Avamar enterprise ready, its deduplication engine has significant limitations. It requires blackout and maintenance windows. With larger capacities, the duration of these windows also increases. With the acquisition of Data Domain, EMC is now focusing more on using its DD Boost technology for distributing the deduplication workload. In fact, EMC recommends the use of Data Domain Boost with Avamar (instead of using Avamar’s dedupe engine) for larger workloads. I believe it was a good decision to support VMware’s SMB market with a technology that was meant for SMB in the first place. I think Avamar dedupe engine is counting its days as a technology that can make money. See my earlier blog on EMC’s backup portfolio.

Stay tuned. More on VDP coming soon!

Client Direct: NetBackup vs. NetWorker

NetBackup introduced Client Direct capability a few years back with NetBackup 7.0 release. This is a break-through innovation in backup infrastructure architecture. Traditionally backup is a process where data is read from production client, transmitted over wire in its entirety to a backup server and then written to storage. The emergence of target dedupe appliances behind a backup server meant that backup can now take three hops through network. It hops from client to backup server first, then it hops from backup server to deduplication appliance. NetBackup changed this game. NetBackup client can dedupe backup stream at source and send deduplicated data directly to NetBackup’s deduplication pool, for example a NetBackup 5020 deduplication appliance, as illustrated below.

NetBackup Client Direct

This architecture is possible in NetBackup because it has several innovations that reduce the impact of running deduplication at the production client.

  1. NetBackup Accelerator: This technology features a platform independent track log that intelligently detects changed files without the need for enumerating the entire file system. Then it optimally synthesizes a full backup image at the storage. The result: Full backups can be run using the resources needed to run an incremental backup.
  2. NetBackup Client Side Deduplication Cache: This enables the production client to run deduplication by comparing the generated fingerprints for the chucks in the changed file (detected intelligently as explained in 1) against the previous backup set without shipping the fingerprint to storage for comparison. The result: Superior federated deduplication without the slow chatter across network.
  3. Intelligent Hybrid Chunking that is not CPU bound: Deduplication chunking is done typically using variable block method or fixed block method. The first one is CPU intensive and the second one is less efficient in data reduction rate. NetBackup uses the best of both worlds by using intelligent hybrid chunking. As deduplication-fingerprinting logic is built into the client, it can start the chunking exactly after identifying the object boundaries. Thus you get the advantage of not being CPU bound while also not suffering from low deduplication rate.

Reducing of impact on production client’s resources, reducing the impact on production network, reducing the number of hops and reducing the impact on backup server (translation, increased scalability) make NetBackup Client Direct a unique feature. The popularity of this feature had made ‘Client Direct’ a common innovation name that appears in RFPs for backup solutions.

The pressure is causing other backup vendors to come up with ‘Client Direct’. EMC announced last week that NetWorker 8.0 will have this capability and even named it ‘Client Direct’ so that the checkboxes in RFP can be ticked. A closer look reveals that NetWorker Client Direct is suitable for checkbox in RFP, but really not ready for primetime as is.

  1. NetWorker Client has no intelligent detection of changed files. NetWorker also does not have any sort of optimized synthetics. The result: Running full backups with NetWorker Client Direct will use significant amount of processing power from production clients.
  2. The NetWorker client and does not federate deduplication; it is done by DD Boost. As these two are essentially unaware of each other’s format, there is no way to cache fingerprints of the chunks from previous backups. That means excessive chitchat with the target Data Domain device during backups.
  3. DD Boost is the process of offloading some of the Data Domain deduplication processing to other systems. In this case, the production clients are taking that load. As clearly documented in Data Domain SISL architecture “SISL takes the pressure off of disk accesses as a bottleneck so that the system relies on the speed of the CPU to deliver inline deduplication performance”. Translation: CPU bound chunking. When this is offloaded to production clients, it can severely affect the performance of production systems with large backup workloads.

Even though EMC can mark the checkboxes in RFPs; their specialists are less likely to encourage POCs with NetWorker Client Direct. In a neck-to-neck battle, it appears that NetWorker has a long road ahead to match NetBackup Client Direct.

© Mr VRay

Page optimized by WP Minify WordPress Plugin