Designing Backup Solutions for VMware vSphere document

For sometime now I’ve been trying to find more information about VMware’s new vStorage API  and while digging around I stumbled across this great VMware document called  “Designing Backup Solutions for VMware vSphere”      

The document in its own words has been written to “Introduce code developers to the procedures necessary to create backup and restore software for virtual machines”      

I know what your thinking….. “Your not a developer !!” ……No not even close but the document did however answer a tone of questions I had about what functionality I might be able to expect from EMC’s up coming release which is going to support the vStorage API.  

If I get time this week I may post listing which functionality in particular I would like to see built into NetWorker.  

The image below links to the VMware document.. enjoy !

        

    

 

     

 

 

    

    

  

 

   

 

  

  

  

 

vStorage Support for EMC NetWorker Coming !

This morning a work colleague attended an EMC NetWorker Webinar and asked EMC’s Anu Yamunan the big question we’ve been wanting to know for quite some time.

Q: IDATA: “When will vSphere vStorage API be fully supported by NetWorker ? As the VCB framework is end of life from 25/07/2010

A: EMC: “We are currently working on leveraging the vStorage API to deliver proxy based protection for VMware environments. NetWorker support for vStorage API is coming soon in the second half of 2010.

I’m really pleased to hear this important feature is in the works, but one particular part of the reply made me nervous “coming soon in the second half of 2010” this is because almost every EMC road map I’ve seen over the years breaks the year up into quarters.

EMC agreed to take this conversation offline and I’m hoping to get a better indication of the release date soon.

Upgrade considerations for EMC NetWorker configured for VCB backups

If your running an older version of NetWorker e.g. 7.3.x and you’ve implemented VMware Consolidated Backups, you need to consider the configuration differences between early and recent versions of NetWorker before upgrading.

Lets take a look at the differences

Pre NetWorker 7.4.1 each each virtual machine was configured as a saveset in the client instance of  the proxy server.

Example shown below.

 

So why is this configuration less then ideal ?

• Firstly the backup server owns the client index and media database information for every VCB client
• Limited visibility of the virtual machines configured in NetWorker
• Both FULLVM and FILE  backup information was stored in the client index

  What changed post 7.4.1 ?

This in my opinion was the point where EMC took a step back and really started to think about how to better improve the configuration of virtual clients within NetWorker.

The image below shows a client configured in NetWorker 7.5.1, no longer is the virtual machine an entry in the saveset field for the proxy server.

 

Why is this configuration better ?

• Each virtual machine is configured in NetWorker as you would any other client (which results in each system having its own client index)
• The FULLVM (full image export) information is now stored in the media database, only vcb file backups populate the client index
• Improved visibility of virtual machines configured in NetWorker

Final Note

This is a good example of why documentation such as the admin guide and release notes should be read before rushing into any NetWorker  upgrade.

VCB and VSS Frequently Asked Questions

If you’ve ever gone looking for information on how the VSS integration in VMware tools works with VCB, then I’m guessing you come up empty-handed.

I remember some time ago reading a document called ” Volume Shadow Service Frequently Asked Questions for VMware Partners ” and thought Id dig it up incase someone found it useful.

Why is VMware offering VSS support ?

The requirement for application consistent VMware Consolidated Backup (VCB) snapshots has been communicated by our customers and partners. VCB will utilize our VSS support to take application consistent VCB snapshots. VSS is also the market standard for providing application consistency for Microsoft applications.

How will VMware provide VSS support?

VSS support will be provided via VMware Tools that run in the guest operating system. VMware will provide a VSS Requestor and a VSS Snapshot Provider (VSP). Changes are needed on the ESX platform as well so moving virtual machines across different versions of host will cause erratic behavior.

The Requester component is available inside a supported guest and responds to events from an external backup application. The Requestor also controls the progress of backup operations inside the guest and interacts with the VMware Snapshot Provider. It will be instantiated upon request of the VMware Tools service when a backup process is initialized.

The VMware Snapshot Provider will be registered as a Windows service and will be used to notify an external backup application of provider-specific events during a VSS backup. The VSP is tightly coupled with the aforementioned Requestor. For this reason, it is only enabled by the Requestor when a backup operation takes place.

 What type of backup is taken in the guest when a VMware snapshot occurs ?

The backup type enumeration is a VSS_BT_COPY backup type. Please see http://msdn2.microsoft.com/en‐us/library/aa384679.aspx for more information on VSS enumerations. This also means that incremental and differential snapshots are not provided by this implementation.

Is there a mechanisim to Vmware tools what type of VSS backup to take instead of a VSS_BT_COPY?

No, in the ESX 3.5 Update 2 implementation this is not possible. In the future, backup partners can have finer control.

What is the sequence of events during a VCB Backup ?

1. The proxy initiates a quiesced snapshot for a VM, it will be able pass the information about whether to create full/incremental/differential backup snapshot and corresponding arguments needed by VSS.
2. The host receives this message and forwards it to the VMware VSS components in the guest.
3. The VMware VSS components in the guest initiates the VSS snapshot based on the arguments passed from the proxy and waits for the volumes to quiesce.
4. The guest notifies the host to create a VM snapshot.
5. The host notifies the guest that the snapshot was created.
6. The guest finishes the quiescing operation.
7. The proxy queries the backup component and writer manifests using the API.
8. The proxy accesses the VM snapshot to perform the backup operation.
9. The proxy initiates a delete snapshot for the VM; it will be able to pass the information about the backup status.
10. The host receives this message and forwards it to the VMware VSS components in the guest.
11. The guest will notify the VSS writers about the backup status so that the logs get truncated only after a successful backup.
12. The host deletes the snapshot of the VM

 Is the VSS manifest made avaiable to the backup application after the snapshot has been taken ?

No, in the ESX 3.5 Update 2 implementation the manifest is not passed to the backup application.

If I wanted to run an ESEUTIL after the exchange backup, How can I do this ?

Using the VCB framework, you can run pre and post operation commands to perform these tasks.

Is the VSS shapshot retained after the backup ?

No, we delete the VSS snapshot during the VMware snapshot operation.

Does the implementation provide any restore capabilities such as revert to VSS copy ?

No, since we we delete the VSS snapshot right away, this is not possible.

NetWorker 7.5.1 new VCB recovery feature

Last week NetWorker 7.5.1 was released on PowerLink, after downloading the package I decided to look through the release notes. Now its safe to say anything to do with VMware and NetWorker interests me so when I saw “Performing A single step recovery of the full virtual machine” I went straight to page 126. If you have a PowerLink account you can download the release notes here.

Below a screenshot showing the new functionality.

 

If your already familiar with NetWorker VCB recoveries you’ll know that to recover a Virtual Machine from a VCB backup is a TWO step process 1. Perform a saveset recovery of the FULLVM saveset and then 2. Use VMware convector to import the virtual machine back into Virtual Infrastructure.

Now with 7.5.1 and VMware converter installed on the proxy server this is now a ONE step process. You can see in the right hand panel you now have fields to enter information you would typically enter into VMware Converter.

I think its really good we are starting to see these kinds of improvements in NetWorker, its not an amazing feature but I think its just a taste of whats to come around the corner.

The following considerations apply when performing a single step restore of a full VMware virtual machine:

• The VMware Consolidated Backup (VCB) proxy system must be running Microsoft Windows 2003 SP1.
• Restore of the full virtual machine is only supported using save set recovery.
• The user must have the required VMware privileges to register and to create virtual machines.
• The VMware converter must be installed on the VCB proxy host machine. If the VMware converter is not installed, the save set of the full virtual machine (FullVM) can be recovered using a traditional NetWorker recovery.
• The VMware virtual machine will restore to the same VMware ESX server orVMware Virtual Center (VC) taken at the time of backup.
• Specifying another VMware ESX server or VMware VC server will cause the restore operation to fail.
• A restore of the VMware virtual machine will fail if the VMware virtual machine already exists in the specified VMware ESX or VMware VC server.

The ESX server must be at version 3.x or later.

Networker 7.5 VCB client configuration

Something that’s been lacking from the Networker Management Console pre 7.5 was the ability to create VCB clients via the configuration wizard, well 7.5 now has this built in !

Ive taken some screen shots and below ill take you through the configuration wizard to show just how easy it is.

Step1. Open Networker management console,  click on the configuration tab from the top menu, highlight clients in the left hand panel and then select the configuration tab from the menu as shown in the screen shot below.

Step1

Step2. Enter the hostname of the virtual machine and tick the “virtual client” check box and select next.

Step3. Next select the option to configure a VCB type backup.

Step4. Here enter the VCB proxy server hostname and select the backup type, I plan to cover the different options in another post but for this demo we will select “Image” for a FULLVM export.

Step5. Next we select the browse policy, retention policy, schedule and have the option to add a comment if required.

Step6. Here we have the option to add the client to an existing group or create a new group, if you create a new group you get the option to configure start time and the option to enable auto start on the group.

Step7. Each VCB proxy server that is not the backup server is considered a storage node, here we have the option to configure which storage node the data gets sent to, keep in mind that if you change this value then all client instances are effected and all data (including traditional agent backups) get sent to this configured storage node.

Note# There is also the option to set the recovery storage node, I really like this option being here as its typically not something people think about.

Step8. And that’s it, here we have a summary page to display the configuration for this client instance.

Networker VCB recovery issue with Linux clients

Ive been performing full image VCB exports of both Windows and Linux virtual machines for quite some time now, I only come across this error when I decided that I would also use the traditional agent based method to backup the file system for some testing I had to do for a customer.

Heres the scenario:

Perform a FULLVM type backup of a Linux virtual machine

Perform a scheduled or manual file system backup of the same Linux virtual machine using the Networker agent.

Open the Winworker GUI on the backup server and try to perform a FULLVM saveset recovery of the Linux virtual machine back to the proxy server and you will see the cross platform error.

And heres the error:

I logged this with EMC and managed to work with one of the engineers on the 7.5 beta team who’s actually issued a fix for the problem. This fix will be included in Networker 7.5 update 1

The only work around currently for 7.4.x systems is to perform the saveset recovery from the command line as the actual bug lies somewhere deep in the Winworker GUI code.