For some time now there have been a number of changes I’ve wanted to see in NetWorker to better support customers with VMware.

Over at nsrd.wordpress.com, Preston de Guise has done a wish list for NetWorker features he hopes to see in 2010 and while we both blog about NetWorker Prestons blog is far superior in terms of NetWorker content, my NetWorker posts these days tend to be mainly focused around virtualization support, so having said that im going to take his idea and do my own version of a wish list.

Dear EMC.

Could you please add to your NetWorker New Years Resolution the following items?

1. To release as soon as possible a version of NetWorker: Which utilizes the new vSphere API’s which replace the VCB framework to allow incremental backups of the FULLVM export using CBT (Change Block Tracking)

2. To improve the way which NetWorker allows clients to be created: When the “Virtual Client” box is checked,  by default NetWorker only allows a client to be configured if it can resolve the name of the system via a DNS  (even an entry in the hosts file will do the trick). This makes life difficult when you want to perform VCB backups of virtual machines which are not configured in DNS and requires an entry in the hosts file to be able to create the client.

3. To improve the way NetWorker maps client name and virtual machine name: To be fair NetWorker has been around for a LONG time and until recent years has always operated using the Server/Client model which relies heavily on DNS in order to be able to connect and perform backups via  the backup agent. Now NetWorker assumes that all clients configured are done so using either short name or the fully qualified domain name, which causes the following problems in a VMware environment.

A. If you configure the virtualization hypervisor support in NetWorker Management Console you’ll see you have a well constructed map showing which virtual machines are configured in NetWorker groups (This is a really great way of identifying which systems are not protected). The trouble here is NetWorker talks to Virtual Center to generate the list of virtual machines and then maps this with clients configured in groups, but assumes the virtual machine name as it exists in Virtual Center  is the same as the ”clientname” as configured in NetWorker which means a number of systems show as not being part of a group.

B. The VCB framework currently allows us to configure two methods for looking up virtual machines, Name (as exists in Virtual Center) and IP, If you leave the setting to the default value of IPADDR: then VCB backups work perfectly, but if you configure the framework to use NAME: then NetWorker once again assumes the client name configured in NetWorker  will also exist in Virtual Center as “clientname” which doesnt work when customers have created or renamed virtual machines using some kind of meaningful description eg. “clientname (sql server)”.

In my honest opinion both these issues could be overcome if when creating a client instance (and having ticked the “virtual client” check box) an additional field could be populated with the name of the client as configured in virtual center and this could only apply when the “VCB” proxy backup type was selected from the apps and modules tab.

4. Allow VCB Backups to be encrypted: NetWorker has a nice feature which allows a password phrase to be configured at backup server level, any client with the encryption directive applied has its data encrypted using AES 256bit encryption before its written to tape. Historically VCB clients with any kind of directive applied would fail and as of NetWorker 7.6 there is only a single supported directive which allows Microsoft Windows system folders to be excluded from backup. In my honest opinion an enterprise backup system should be able to encrypt all backup data.

Ive just recently sat and passed the VCP-410 exam and thought Id post my thoughts after the fact.

The exam overall is probably a touch harder then the VCP-310 exam, I felt there were more questions where multiple choices were required, and im sure the number of questions has increased from 70  to 85.

Having taken the exam now, I can say I dont think the 2 day “whats new” course is quite enough to prepare you for the exam so make sure youve spent a fair amount of time familiarizing your self with vSphere and touched up on the basics.

Ill briefly summarize the documentation I used to prepare below, but if you’re looking for a more comprehensive list, then you should check out Simon Longs recommendations on material here, hes even created some online practice exams which are really good.

iSCSI SAN Configuration Guide : Make sure you read up and understand how iSCSI differs from Fibre Channel and understand how CHAP fundamentals.

Fibre Channel SAN Configuration Guide : Make sure you understand the concepts of LUN masking and also the requirements for booting off a SAN volume.

vSphere Licencing Guide : Make sure you know what changes have been made in vSphere around licensing and how this affects datacenters with a mixture of ESX 3 and ESX 4 systems.

High availability : Like I said early in the post, brush up on basics as questions which involve HA are not vSphere specific and you should definitely have a good understanding of how HA works.

Resource Management : Now resource management covers both DRS and Resource Pools, once again these questions where not so much based on vSphere but more on good old basics. Make sure you have a good understanding of Resource Pools and fairly familiar with reservations and resource shares.

Configuration Maximums: Definitely check out this document as there were a tone of questions based on the information in this document.

Summary

Out of the 85 questions in this exam I only remember twice thinking to my self, “ah thats a bit of a tricky question“, overall I think VMware want you to pass this exam and what I mean by this is the questions are nothing like the you would find in a Microsoft exam.

Also something that has been well covered by a number of blogs is the VMware Certified Professional 2nd Shot/Upgrade promotion which gives you a free second shot at the exam if you fail the first time around. Follow the link to register.

If you are an existing VCP on VI3 then you only have untill December 31st to take the VCP-410 exam without needing to attend one  of the courses listed below.

• If you are NEW to VMware
  • Attend the VMware vSphere 4: Install, Configure, Manage course OR attend the VMware vSphere 4: Fast Track
  • Take and pass the VCP on vSphere 4 exam
• If you are currently a VCP on VMware Infrastructure 3
  • Take and pass the VCP on vSphere 4 exam.  This option will only be available until December 31, 2009.  Beginning in 2010, VCPs on VI3 must attend the VMware vSphere 4: What’s New class in order to upgrade.
• If you are currently a VCP on ESX 2.x
  • Take and pass the VCP on VMware Infrastructure 3 exam
  • Take and pass the VCP on vSphere 4 Exam. This option will only be available until December 31, 2009.  Beginning in 2010, VCPs on VI3 must attend the VMware vSphere 4: What’s New class in order to upgrade.
• If you are not a VCP on VI3, but have attended one of the prerequisite classes (Install & Configure; Deploy Secure & Analyze; or Fast Track).
  • Take and pass the VCP on VMware Infrastructure 3 exam OR attend the VMware vSphere: What’s New course.Take and pass the VCP on vSphere 4 Exam.

By far one of the more popular posts Ive done over the last few months has been this post where I showed how the iSCSI initiator in vSphere 4 could be configured to provide multiple paths to each LUN and in turn the path selection policy “Round Robin” could be configured to load balance across multiple paths (up to 8 paths is supported).

One of the only things to disappoint me with the initial vSphere release was this configuration could only be applied to a standard vSwitch, when trying associate the VMKernel ports on a dVSwitch with the iSCSI initiator the following error occurred ” Add Nic Failed in IMA“

Over the last couple of days Ive been itching to test this out and last night I finally got a chance. Also something I wanted to do was perform these tasks using the vSphere CLI opposed to how I did it last time which was via esxcli on the service console.

Ill skip through the part where I created the dVSwitch, but the same concepts apply, 2 nics, 2 port groups. In my test lab VMK1 and VMK2 were the VMKernel ports associated with the port groups as shown in the screenshot below.

Step 1.

Associate VMK1 and VMK2 with vmhba33

Step 2.

Now lets check to make sure that both VMK1 and VMK2 are indeed associated with vmhba33

At the time of writing this post I dont have a iSCSI system I can easily point my vSphere system at to confirm this works but the fact that I can now successfully run the commands without error is promising.

Ill setup a Celerra Simulator over the next day or so and confirm everything works as expected.

I would be interested in hearing from anyone who’s also got this up and running.

Just as a wrap up, heres a couple of errors you might come across and how to fix them.

Get Hba Oid Failed error when trying to add vmk’s to the iSCSI initiator, this was because the iSCI initiator was disabled.

If you’re using vSphere 4 Update 1 and your still getting error Add Nic Failed in IMA then its likely because you have not configured the fail over of the physical NICS correctly.

Myself and colleague Preston de Guise recently performed beta testing for the latest build of EMC NetWorker, I can only speak for my self but I was quite surprised by the  lack of regressed bugs I found and overall it’s looking very good.

I was also hoping  NetWorker 7.6 was going to show that product development teams were still expanding on NetWorker’s already existing support for virtualization and also listening to what partners and customers were asking for….Im glad to say that 7.6 has a couple of new features which I think have shown exactly that.

Hypervisor Support

This was first introduced in 7.5 (I wanted to cover this as there have been changes). NetWorker now allows an administrator to configure a single or multiple virtual center servers which allows NetWorker to display a map or table view of the VMware Infrastructure.

7.6 has added a nice feature which now incorporates the VCB Proxy server into the map, below is an example of this from my test lab.

from left to right we have vSphere host, Virtual Machine, NetWorker group showing Virtual Machines within and then to the far right we have the vcb proxy server.

The great thing about this feature is it gives you a detailed map showing which virtual machines are configured in NetWorker groups.

One of the greatest things about virtualization is we no longer talk about days and weeks to provision a new system, now we talk about minutes, especially when deploying from a template. While this has made life a tone easier for us, Ive seen many times situations where people would  create virtual machines willy nilly, put them into production and forget to notify the backup administrator of A. The existence of the virtual machine and B. The backup requirements.

This NetWorker feature makes it very easy for the backup administrator to report on virtual machines not configured in NetWorker groups.

New 7.6 VCB Functionality

Only VCB Image backup required to perform Image based or file based recovery: Now with 7.6 NetWorker can perform file level recoveries of the file system inside the VMDK file from the FULLVM backup (Windows Only) This means we no longer need to have additional groups or client instances with VCB file system savesets configured.

Directive: There is a  new VCB Directive, this when configured against a VCB client instructs networker to skip the backup of the following windows system folders, Windows\System and Windows\System32

VCB Framework Settings: Traditionally all VCB framework settings were configured in the config.js file on the proxy server, now in 7.6 this information is now stored in the proxy servers ”Application Information” client resource field. This helps to make as much of the proxy configuration visable within NetWorker as possible.

The following example displays all the possible attribute values used for VCB configuration

VCB_HOST=any.vc-or-esx.com
VCB_BACKUPROOT=G:\mnt
VCB_TRANSPORT_MODE=hotadd
VCB_VM_LOOKUP_METHOD=ipaddr
VCB_PREEXISTING_MOUNTPOINT=delete
VCB_PREEXISTING_SNAPSHOT=delete
VCB_MAX_RETRIES=2
VCB_MAX_BACKOFF_TIME=15

Single Step Recovery: Although not new to 7.6, this was introduced in NetWorker 7.5 and greatly simplifies the process of recoverying a virtual machine. Tradidtionally in prior versions, you would need to recover the FULLVM saveset back to the proxy server and then use VMware Convertor to import the machine back into virtual center. With single step recovery you just populate the fileds as shown in the screenshot below and let NetWorker do all the work. The only prerequaite for this to fuction correctly is VMware Convertor 3.0 or above must be installed on the backup server.

Notifications

Ive already mentioned NetWorkers ability to display map and table view’s of your virtual infrastucture to help identify unprotected virtual machines, but as of 7.5 and also included in 7.6 is a new notification which alerts when new virtual machines are created in your environment, but not configured in an active NetWorker group.

Ill be posting some additional information and screenshots from 7.6 testing over the next couple of days so be sure to check back.

When creating iSCSI LUNS on a Celerra I typically have all file systems created before hand and then use the command line to create iSCSI LUNS in each file system.

A couple of weeks ago I decided for some reason to use the wizard “Create new iSCSI LUN” and noticed something that was not present in the older versions of DART.

Heres a screenshot showing the option to virtually provision the LUN

This seems to have come in with DART version 5.6.45 as is not present with the latest version of the Celerra Simulator which runs 5.6.43

For those of you who are interested in how to create LUNS from the control station, heres a couple of examples.

Create a virtually provisioned 100GB iSCSI Lun using LUN ID of 5, configured on iSCSI Target “ProdTarget” inside file system “ProdVMFS”

server_iscsi server_2 -l -number 5 -create ProdTarget -size 102400M -fs ProdVMFS -vp yes

If you’re creating destination “ReadOnly” LUNS for iSCSI replication then we use the same command but add a switch to configure the LUN as read only.

server_iscsi server_2 -l -number 5 -create ProdTarget -size 102400M -fs ProdVMFS -vp yes -readonly yes

I was doing some work on a couple of EMC Celerra’s this week and found an issue which I expect is a bug in the DART code, I managed to work around it using the command line, so I thought I would post incase anyone else comes across this same issue.

Initially I had fully provisioned LUNS on Both Celerra’s but decided in order to conserve some file system space,  tear everything down and recreate the LUNS on the DR Celerra using virtual provisioning, this greatly reduces the amount of file system space needed to allow replication of iSCSi LUNS to occur.

Here is the error I got while running through the replication wizard

The error here is basically saying ” I can’t find any suitable iSCSI LUNS to replicate to” which is sometimes the case when you forget to do one of the following two things.

1. Ensure source and target iSCSI LUNS are identical in size

2. Ensure the target LUNS have been configured as “READ ONLY” LUNS.

I knew it was neither of these two things and after a couple minutes of thinking about it,  the only thing to have changed was the LUNS at DR were now virtually provisioned.

After a little bit of thought I decided that I would try to create the replication task from the control station and sure enough the command completed successfully and the LUN replicated without any issues. 

Hers the syntax I used

nas_replicate -create LUN7 -source -lun 7 -target iqn.1992-05.com.emc:ckm000850009000000-1 -destination -lun 7 -target iqn.1992-05.com.emc:ck2000800009760000-2 -interconnect SRMReplication -source_interface ip=192.168.0.10 -destination_interface ip=192.168.5.10

Also if you are replicating between two Celerra’s, make sure that both are running the same version of DART and your running at least 5.6.45 or later.

*****UPDATE*******

This issue is resolved in DART 5.6.45

Vmware vCenter Linked Mode enables a common inventory view across multiple instances of vCenter Server.

Over the last year Ive deployed a number of VI3 systems with Site Recovery Manager in the mix and one of the things that always annoyed me was having to manage each site separately. Now with vCenter Linked Mode both sites can be managed from a single view. (Including all Site Recovery Manager functionality)

Something I found common on the forums was VMware customers using Virtual Center “Standard”  edition at the production site, but purchasing “Foundation” for the recovery site as it was considerably cheaper.

Now where am I going with this ? Lets take a look at one of the images I found in one of the Vmware documents I had laying around.

Ahhhh … no Linked Mode with vCenter Foundation.

Not everyone will think this is a big deal…. personally, having a single view for managing multiple vCenter sites is brilliant, its defiantly worth considering going with at least Standard at the recovery site for this additional feature.

Yes, much to my surprise today SRM 4 is now available from the VMware download site.

I was at a customers site this morning when an email from VMware come through on my BlackBerry announcing SRM 4 had been released, After racing home at the end of the day I logged in and started to download the new SRM 4 package and Celerra Storage Replication Adapter. Look at that pitiful download speed, it was like watching paint dry.

I havent had a chance to read over the release notes properly yet, but here’s whats new.  Keep posted as ill be posting more about SRM 4 over the next few days.

Also if your interested in the documentation, here are the links to the Admin Guide and Compatibility Guide

New in This Release

• Full compatibility with vCenter 4.
• Full support for NFS-based arrays.
• Support for shared recovery sites.
  Enables many-to-one pairings of protected sites with a recovery site. For more information, see the technical note Installing, Configuring, and Using Shared Recovery Site Support, which is available at http://www.vmware.com/support/pubs/srm_pubs.html.
• Resilience in the face of vCenter unavailability during a test recovery.
  Placeholder virtual machines can be quickly repaired after the protected site vCenter becomes available again.
• New repair-mode installation features.
  You can run the SRM installer in repair mode if you need to change configuration parameters such as vCenter credentials, database connection information or credentials, and certificate details.
• Graphical interface to advanced settings.
  Eliminates most requirements to edit the XML configuration file
• Support for DB2 as an SRM database server.
• New licensing options.
• Improved scalability.
  A single protection group can now include up to 1000 virtual machines.
• Full Compatibility With DPM (Distributed Power Management)
  SRM recovery plans can now power-on or power-off a host that is in standby mode.
• New Option to dr-ip-customizer Utility
  The dr-ip-customizer utility now logs less verbose diagnostic output by default. To force dr-ip-customizer to log the same level of diagnostic output that it produced in earlier releases, use the -verbose option.
• Change in Certificate Validation
  When you select certificate authentication, the SRM installation validates the certificate you supply before continuing. Certificates signed with an MD5 key are no longer allowed.
• Support for Protecting Fault-Tolerant Virtual Machines.
  SRM can now protect virtual machines that have been configured for fault-tolerant operation. When recovered, these virtual machines lose their fault tolerance, and must be manually reconfigured after recovery to restore fault tolerance.
• Improved context-sensitive Help.
• PDF documents available on release media
  Current versions of the PDF documents for this release are available in the docsfolder at the root of the SRM 4.0 CD. Updated versions of these documents may be available at http://www.vmware.com/support/pubs/srm_pubs.html.

Ive had Replication Manager 5.2 integrated with VMware VI3, hanging off an EMC Celerra using iSCSI for some time now, and ever since vSphere was released Ive been meaning to test the functionality to make sure everything works and to see if there are any changes.

Now having set this up with 3.5 update 4 hosts I remembered one of the key steps is changing the advanced LVM.EnableResignature option to 1 which allows a snapshot of an existing lun with a matching header to  automatically re signature and be presented back to the host. If you want to read more about how this works then Chad Sakac has a really good post about this here.

 Here is a screenshot showing this on a ESX 3.5 host.

The next step was to build my self a vSphere 4 host and integrate it into my existing lab setup, after building the host and searching through the advanced options I realised the LVM.EnableResignature option was not available and after a quick google it didnt take long to find this post by Duncan at Yellow-Bricks.

After configuring the host in Replication Manager I performed a Mount of an existing snapshot to my vSphere1 host, the task completed successfully but I was unable to see the lun on the host.

The image below shows the snapshot has been succesfully mounted to the host.

You can see only the default datastore on local disk and the original Celerra Lun is shown.

Next I went to Configuration >> Storage >> Add Storage >> Disk/Lun and there it was.

 After selecting the lun and clicking next, I was now presented with three options as shown in the screenshot below.

As Duncan pointed out in his post, yes this can be done through the GUI but its more fun from the command line, im also a firm believer of learning to do tasks from both the command line and the GUI.

So with that said im now going to use vicfg-volume to resignature the lun.

First lets check the existing header ID  using vicfg-volume.pl – -server vsphere1 -l

Now lets resignature the lun using vicfg-volume.pl – -server -r <existing_header>

Next I check to see if the lun is now visible, and there it is.

Just as a last note to this post I just wanted to mention Ive not yet found anything in the release notes to confirm vSphere is supported with Replication Manager 5.2.2, so at the moment this is nothing more than an informational post. Ill make sure to update this as soon as Ive confirmed this is in fact supported.

Oh and just incase you’re wondering why this change has come about…. With VI3 and LVM.EnableResignature it was an all or nothing setting, now with vSphere 4 you can change this on a per Lun basis, actually a good thing once you know about it.

If you’re a VMware shop you have to check out some of the demos online, Replication Manager is a Brilliant product !

Something I noticed last week while working on a customers site was multiple iSCSI initiators logged into iSCSI targets which I had not granted access.

I’ll fire up my lab to demonstrate what I saw and how to change the behavior.

Here on my Celerra simulator I have three iSCSI Targets configured for VMware, Exchange and SQL. Each Target uses a different interface with its own IP address.

Im going to use my Exchange Virtual Machine to demonstrate, but before I flick to the VM, I want to show the mask for the VMware and SQL iSCSI targets.

VMware -Only the vSphere initiator masked here.

SQL - Only the SQL initiator masked here.

Now lets flick to the properties of the Microsoft iSCSI initiator in my Exchange VM. You can see Ive only configured the IP address assigned to the Exchange iSCSI target.

Now lets take a look to see what targets are visible.

And here you can see the iSCSI initiator is able to see all three targets. The reason for this is because by default the Celerra returns information on all targets to all initiators regardless of the lun mask.

Although the systems can only access the luns the mask permits, it’s still quite messy when you have 5, 10, 15 targets on the Celerra which are visible to all Microsoft initiators. So luckily the guys who write the code have a parameter for us to change which alters the default behavior.

First lets confirm the current setting.

[nasadmin@csprod ~]$ server_param server_2 -f iscsi -i SendTargetsMode
server_2 :
name                    = SendTargetsMode
facility_name           = iscsi
default_value           = 0
current_value           = 0
configured_value        =
user_action             = none
change_effective        = immediate
range                   = (0,1)
description             = Enables return of information about inaccessible targets

Now lets change the current value to 1

[nasadmin@csprod ~]$ server_param server_2 -facility  pre=”-facility “>iscsi -modify SendTargetsMode -value 1
server_2 : done

And now confirm the current vaule = 1

[nasadmin@csprod ~]$server_param server_2 -f iscsi -i SendTargetsMode
server_2 :
name                    = SendTargetsMode
facility_name           = iscsi
default_value           = 0
current_value           = 1
configured_value        = 1
user_action             = none
change_effective        = immediate
range                   = (0,1)
description             = Enables return of information about inaccessible targets

So what Ive  done here is configured the Celerra to only return information on iSCSI targets when luns have been masked to the client initiator.

Restarting the iSCSI service on the Celerra causes the initiators to drop the targets to which it has no luns maksed (Actually do this with caution as it causes all iSCSI connections to drop)

Lets take a look at  my Exchange VM after this has been done.

Much Better ! If you dont use the Microsoft iSCSI initiator than this is not really going to bother you, VMware’s software initiator only shows targets when connected to luns.

In my honest opinion this should be the default setting.