Backing up Exchnage 2007 and SQL 2005 using VCB

[asn1-bur]

Afternoon All,

I have just upgraded my ESX cluster to 3.0.2 and would like to open up a discussion on using VCB to backup Exchange 2007 and SQL 2005 servers.

I have read in this forum and various other sites of some success with backing up exchange using the freeze and thaw scripts but I would like to hear from someone who is using this with Exchange 2007 and SQL 2005 in the production environment.

My plan is to do nightly backups of Exchange and SQL using our backup software (CommVault) and then do a week end fullvm backup of the servers Exchange and SQL server using vcb, cloning them up to a LUN that I have presented to my backup server and then backing them up to tape, the hope would be that this would put me in good stead in the case of a disaster.

Appreciate any feedback on this strategy (positive or negative), any thoughts on a better way of doing things and also from anyone that has success with this method backup up Exchange 2007 and SQL 2005 with VCB.

[Rumple]

When it comes to transactional systems like SQL, Oracle, or Exchange I typically will get the Operating system drive imaged (for a bare metal recovery), but use API compliant tools like ntbackup, SQL maintenance plance, RMAN (or any commercial backup products) to get a consistent supported backup of the data.

In the case of a disaster I just need to restore O/S Drive and add a couple blank drives ot match the original system. I then re-create the directory structure that was present before (ie, for exchange the dir holding the transaction logs and then the directory on the data drive for the mailstores).

Once that is done I can then just run an exchange restore and its back up and running.

Same thing for the other databases.

[asn1-bur]

Thanks for your response.

So when you talk about getting the operating systems drive imaged are you talking about using VCB with the Fullvm option or are you actually talking about using some imaging software such as ghost?

It seems that your approach is very similar to what I am trying to achieve.

[Rumple]

For the VM's primary drive I would use either Vizioncore Ranger or PHD esXpress.

esXpress has a free version which does only full backups..but since you only really need to image the C: drive periodically (and use system states and integrated backup software for the data) it doesn't matter.

[asn1-bur]

Is there a reason why you use either of these products instead of vcb to capture an image the drive? I assume that they handle transactional systems better than vcb?

[Rumple]

I believe they both integrate with the vcb to allow you to do a full lan free backup, but I don't think vcb will quiese the drive (its basically a crash consistent backup) but maybe thats changed in later versions.

When backing up without vcb you can tell Ranger (and I'm pretty sure esXpress) to quiese the drive and then perform your backup.

In reality though, even a crash consistent backup isn't too bad as long as the system will actually boot into safe mode afterwards. Once its booted you'd then perform your data restore including the latest system state and then you are back to a consistent state anyhow.

I haven't used vcb yet as I just got my backup software to a level to suppost ESX and vcb but I can't upgrade the backup server host to Windows 2003 as I have to rebuild vs an upgrade so my vcb information may be flawed.

Can anyone else with more knowledge respond...

[kix1979]

Keep in mind that the quiesce of the file system does NOT quiesce transactions that are in memory. That means that any transaction that is in the middle of writing to disk will not be completed successfully and will get dropped if you have to do a restore. Also the quiesce of the file system can and will drop your database, so don't use it on a VM with a database.

If you want to quiesce a database within a VM for backup you have a few options: write your own script, use an agent, or use VSS. For esxRanger Professional we chose to go the route of creating our own VSS driver to quiesce the database writers in the VM (VSS does imply Windows, Linux is very different). This way you are 100% consistent and all current writes to disk are flushed and future writes are paused for a microsecond. The same can be said by using a database agent or writing your own script, they will be able to provide the same consistency, just for different levels of availability and management.

At the end of the day, can you live without quiescing a database? Sure, but do you want your job to be on the line if you lose data in that production database? Because you will lose transactions if you don't quiesce.

Kix @ Vizioncore

[asn1-bur]

Thanks for both your input.

My aim here is to get myself in the best possible position for recovery in case of a disaster, i currently perform full backups of the Exchange and SQL servers using CommVault Galaxy (using the specific SQL and Exchange agents), backing up the machines as if they where "real" machines so I am happy that I have good recoverable data (multiple DB restore tests have proven this).

However, what i want to avoid in case a full system recovery is having to rebuild the VM from scratch and then restore the data.

Therefore my plan was to combine full daily backups using commvault with image (.vmdk) level backups of the VM using VCB. This in theory would allow me to restore the VM from the VCB backup and then restore he database from the Commvault backup.

In this situation would the fact that VCB doesn't quiesce machines with transactional DBs very well really matter as long as it gets me in the situation where I can restore the VM and then resotre the DB?

[kix1979]

It would not be as big of an issue in that case, since you are supplementing the Image backup with an agent based backup inside the guest OS to quiesce and backup the tables. However, make sure on those VMs you REMOVE the Legato Sync driver. It's part of the VMware tools install and will run every time you use VCB, there is no way to disable it other then removing the driver.

[asn1-bur]

Thanks, I will do (remove the sync driver).

Overall do you think that this is a solid strategy or do you feel that it could be improved?

[kix1979]

How big of an environment are you planning on backing up with VCB? I know lots of folks that use them, obviously I am a little biased since I work for the competition, but it will work just fine.

I would still like to see that you do quiesce the file system and any other writers before you do the image backup. This can be done manually per VM or with script/3rd party tool. The reason being, it's on more level of redundancy you can count on. Worst case with a tape is you cannot get it or it's dead, so having the database in a state that it can be used with known good data is a plus. Again, it will still work just fine without this, but I always look at how to mitigate risk as much as possible. Especially on email, the non-critical application that costs IT people their jobs...

Kix

[asn1-bur]

Sorry for the late response on this. I am looking at this for one of the satellite offices, only 100 users or so.

So to confirm VCB does not do any quiescing? I was under the impression that it did???

[asn1-bur]

OK I lost the plot a bit there with my previous post. From my understanding VCB does quiesce the file system but not SQL or Exchange DBs.. correct... right?

Have you had any luck with the freeze and thaw scripts for exchange? Is there something similar for SQL?

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=5962168

[dinny]

When VCB starts a backup job of a VM (either file or full) a snapshot is created - as part of the creation of that snapshot the Vmtools running on the VM (presuming vmtools is installed and running) will quiesce the file system (by default on Windows VMs).

(This default behaviour can be disabled by uninstalling the file system driver that does the quiescing)

This file system quiescing can cause issues with databases.

You can disable it - but the DB would still not necessarilly be backed up in a

data consistent manner - neither would the file system.

Better to use a custom pre-freeze and post -thaw scripts to stop Exchange/SQL before VCB takes the snapshot and start them again once the snapshot is complete.

(This would obviously mean the backups would need to be scheduled outside the servers SLA)

The pre and post scripts could be as simple as a couple of batfiles containing some net stop xxxx[/i] and net start xxxx[/i] commands.

Or you could write a couple of more sophisticated stop and start scripts in vbs or something.

Dinny

[asn1-bur]

For SQL would just stopping the SQL Server (MSSQLSERVER) service be enough?

[dinny]

There is usually a dependent service - sqlserveragent - but I suspect stopping the sqlserver service will automatically stop the agent service too.

It'll be easy enough to test...

Dinny

[asn1-bur]

I'll go ahead and test tomorrow on SQL but I think stopping the service shoudl do the trick!

One last thing and then I think i'm done on this. If I use the exchange and SQL freeze and thaw scripts I am not clear on if I should be using the VCB SYNC driver or not, I do not currently have it installed, but when I read the text below it seemed to say to me that without it I would only get crash consistent backups, my current thinking is freeze and thaw with the sync driver, is this correct?

"Disabling the SYNC driver allows you to keep the Exchange Server on-line, but results in snapshots being only crash-consistent"

"Use the following pre-freeze and post-thaw scripts to take Exchange Server offline for approximately 60 seconds and then restart it again after the snapshot is taken. This approach leaves Exchange Server inactive, but keeps the SYNC driver enabled while the snapshot is taken, ensuring application consistency. Using this method, you create the quiesced snapshot of Exchange Server"

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=5962168

[dinny]

That's the logic I use.

I leave the sync driver installed - and just stop the DBs with the pre-freeze.

That should leave the image file consistent and DB consistent as I understand it.

I tend to just do a weekly image backup - then use the native SQL server backup jobs to back up the individual DBs to a flat file.

Then I can do daily file level backups of the backup directory - allowing me to do restores of individual DBs and not just the whole server.

You just need to ensure that all backups are scheduled out of hours - as the pre-freeze script will kick in on all backups - including the file level ones.

Dinny

[matt-brewster]

The pre-freeze and post-thaw scripts can start/stop services when the snapshot is being taken, but do we need to do anything when VCB deletes the snapshot later on, after it has finished?

Will snapshot deletion interfere with the Exchange / SQL databases?

[dinny]

I've never noticed an issue - but my databases aren't particularly heavily used and the snapshot deletions are normally out of hours too, as none of my VMs are that big.

I have read other VMTN posts suggesting that some people notice a second or two of unresponsiveness on a VM when a snapshot is taken or committed - it's not really been an issue for me though.

I guess if you want to avoid snapshots totally you're just left with options such as running backup agents in the VM and doing file level backups across the network - which also has its down sides....

Dinny

[asn1-bur]

Fully tested SQL today in the lab with VCB backup using the freeze and thaw scripts and all signs are very positive.

Found a problem with exchange 2007 though, it looks like the sync driver isn't supported on x64 bit windows.......

[asn1-bur]

I can't find anything saying that the sync driver will be supported on windows x64, I guess i'm sol for now...

[c0de1]

VMware Tools. The SYNC driver is currently not supported on the following operating

systems:

! 64‐bit guest operating systems

! Any operating system other than Windows

[Erik_Zandboer]

Yes it would. I use esXpress myself, they have an excellent option for quiescing. They can use the VMware Tools for quiescing. However, I do not use this option for SQL or Exchange. I just have esxPress (soft) shutdown the VM, create the snapshot, then power back on. Works every time again, you are very sure you have a consistent backup (it is a backup of a shutdown machine!). Off course the down side: Your VM has to be shutdown before backing it up. As long as it is no problem that your VM picks up 5 minutes of downtime every time you run a backup, it is a very certain way of making backups.

In my environment I have less than 5 minutes downtime per VM every night (the backup runs after the VM has been restarted so downtime is minimal)

[rscarinci]

Erik, does the size of the VMDK effect the amount of time the VM needs to be down for backup with esXpress?

[PerryWhittle]

depends if you make a delta or full backup, esXpress is very efficient at backing up VM's