No point in locking the door when walls have fallen

Security for information assets is crucial for business continuity. Corporate information in the wrong hands can compromise the survival of an organization which is why security must be considered wherever that information lives. The last stop for the protection and recovery of this information is backup.  It protects against the loss of information from hardware failures, logical corruptions and human errors.  Unfortunately, the security of backup information can often be overlooked. As a leader in backup and security, Symantec advocates a holistic approach to managing and securing corporate information.

Let us consider a few of the key components for secure enterprise backup solutions.

  • The solution must securely transfer data from source to backup storage.
  • The solution must store the backup image securely.
  • It should offer authentication and authorization built-in to control access.  You don’t want an intruder or a client system masquerading as a production client and retrieving data from backups.
  • It must protect itself completely. If the system hosting the solution suffers a hardware failure, it must be able to recover image metadata, security certificates, access control rules and other important data so that the solution once again controls access to backup images.

While there are more considerations that could be listed, these underscore the broader requirements for an enterprise backup solution.  Backup is much more than moving data as quickly as possible from source to backup storage.  Other functions need to exist to ensure the security of backup images and to prevent the production data embedded in a backup image from falling into wrong hands.

Moving data to the cloud is a great use case to consider in more detail. In an era where organizations are looking to cut operating costs by moving information to the cloud, attention must be paid to the protection of user data from end-to-end, regardless of whether it is on primary storage on a VM hosted for the user or sent to a backup storage.  Choosing a backup solution that fails to deliver security for backup images may result in corporate embarrassment, liability and loss of business.

I came across an interesting post from Mike Beevor who works for Veeam as a Systems Engineer. You can read the details here. Mike creates a nice article by consolidating the scripted responses needed when an IT security team is evaluating the risks in using Veeam for VM backups in a secured environment. Unfortunately, he looks to have left one huge hole unattended. If a tornado knocks down the walls, is there any point in putting locks on rest of the doors?  Let me explain what I mean.

Locking the front door when walls have fallen!
Locking the front door when walls have fallen! Illustration by Scott G.

The failure point in question is the Veeam Backup File(VBK). When production VMs with precious data are backed by Veeam, it stores virtual machine files in a container file with extension vbk . This file is kept on a plain file system on a backup server with direct attached storage, SAN attached array or on a NAS device.

Most production VMs will have one VMDK file for operating system and one or more VMDK files for data. A utility was originally developed to provide users with a way to import a VM using a VBK backup file directly into vSphere. Veeam created this process because the backup solution does not offer a good way to protect itself (see rule number 4 – protect the backup & recovery system).  A person who gets a copy of a VBK file can import the VM in the file onto his/her own ESXi host, detach the data disk and mount it on his/her own VM to get access to production data! Veeam does not provide any sort of encryption for VBK files.  Unfortunately, the only way to recover individual objects from a Veeam backup is to run the entire VM from backup storage.

The lack of security for the container file makes it easy for anyone to retrieve data. The users of Veeam are already concerned about this weakness. Posts in the Veeam forums related to this issue seem to be conveniently moderated out. Here is an example of a post which appeared in a Google search before it was deleted for Veeam 6 launch. Finally a modified response appeared that Veeam will consider this as a future feature.

 

Customers requesting enhanced security for VBK, moderated thread reappeared recently

The user is asking for enhanced VBK security through the use of password protection.  It is a step in the right direction. Hopefully, Veeam will work on this soon.

As you can imagine, this is a huge security hole especially as users virtualize more mission critical applications.  Unfortunately, Veeam 6 makes this problem even worse. Now these VBK files are scattered around multiple repository hosts thereby increasing the chances for exposure.  

What to do if you are currently using Veeam for backups?

  • Enable file system level encryption on all repositories if overall performance after encryption is acceptable.
  • When using a NFS/CIFS based deduplication device for storage (e.g. Data Domain), enable encryption within the device.
  • Make sure that the NFS/CIFS shares are exposed only to proxy servers and Veeam servers. NAS devices’ default export policies are generally read-only access for ‘world’. In the case of VBK files this read-only access is enough to compromise production data in backups.
  • Harden passwords on all Veeam backup servers, repository servers and proxy servers.  Do not use the same password on all repository servers.
  • Talk to your security team; leverage their investments. For example, the security team may have Symantec Critical System Protection suite.  Install CSP agents on backup servers and repository servers to provide non-signature based Host Intrusion Prevention protection. It protects  against zero-day attacks using granular OS hardening policies along with application, user and device controls
  • Consider switching to a backup solution that offers encryption for both data in-flight and data at rest.  In fact, you may already be using another backup solution to backup Veeam backup files. Most backup applications offer vADP integration with VMware. Do you know that Backup Exec is #1 in Veeam backup? Veeam’s Doug Hazelman admitted this to Curtis Preston.

Veeam has done similar things in the past to conveniently hide the root of the problem with deflection techniques. Remember DCIG’s Jerome Wendt who uncovered the real motto behind SureBackup? Learn more about his discovery here.

Naturally, the next question is how a leader like Symantec provides security for backups.  Let us use Symantec’s Backup Exec as an example.

 

  • When making use of deduplication-folder (Backup Exec’s built-in deduplication), it is not possible for an intruder to identify specific backup images even if he gains access to the file system where the folder resides. If he decides to steal the entire folder, he cannot import the images on this folder to an alternate system without having the credentials needed to access this deduplication folder.
  • When sending backups to tape, software and hardware encryption (T10 encryption standard) are supported. Thus you do not have to worry about information getting leaked even if tapes are stolen.
  • Backup Exec uses security certificates between clients and media servers. It is not possible for an intruder to masquerade as a client and request a restore of production data.
  • Self-protection: Backup Exec not only protects the production data, but it has the capability to protect itself against hardware failures or human errors.

 

Leave a Reply

Your email address will not be published. Required fields are marked *