Anyone else having these VDR issues?

[jarsenea]

I love the fact that VMware is providing VDR as part of the vSphere package. It's definitely a step in the right direction, albeit I'm still inclined to think this software hasn't been put through the ringer in terms of proper QA. I'm just trying to put out a feeler to see how many others have experienced some of the same issues I'm having.

To start, I'm backing up my VMs via a network share on a standalone Windows 2003 server that has a NAS attached to it.

Some of the issues I've noticed:

1) Backups take an inordinate amount of time. I can understand the first backup, but my VMs don't change very much from day to day. Most of the data being manipulated is located on RDMs are these are backed up using Tivoli, not VDR (I use VDR solely for the OS partitions). Each partition is approximately 25GB, there are 15 VMs and my backup window (10pm - 6pm) isn't sufficient to complete the process.

2) Integrity checks for the backups are taking a crazy amount of time and will usually stop due to my window being closed (see point #1)

3) I'm getting inconsistent "failures" for certain VMs (the report will simply state that a VM failed to backup, not much else). It also varies per night and not always the same VMs (not exactly sure if this is related to #1 where the window is closing while VDR is executing)

4) I had the most difficult time setting up the remote share from the VDR appliance in vSphere. The username and password would never be accepted (even though if I tried the same share with the same user/pass on a Windows machine, it would work fine). I finally narrowed down the problem to the simple fact that the VDR appliance can't handle passwords that have special characters in them (this password had an "@" and a ","). Looking at the console while attempting to mount the share would spit out a CIFS error -22. Changing the password to include only numbers and letters was sufficient to work around this issue.

5) Snapshots not being created for no apparent reason and thus failing the VDR process. I'm fully able to do a manual snapshot with or without the memory state, so I'm not sure why VDR can't do it. This issue is very intermittent. I had it often when I first setup VDR, but now it only happens every so often (without any type of consistency).

I think that's all I can think about for now..

[pharmer]

Absolutely the same issues here:

1. Constant auth issues accross various clients

2. Timezone hacks required to get the appliance time to work properly

3. Integrity check unusably slow, often fails and tends to create corrupt restore points

4. Random snapshot failures, even though I can manually snap and all DNS is configured correctly

And to top if off - no backup success/failure notification

Very poor even for a 1.0 product

A very disappointed enterprise customer here with regards to VDR

[TAMW]

I totally agree,

VDR 1.0 was really a disappointment for me.

VDR 1.0 is imho not ready for production use, it looks more like a technical preview than a RTM release.

[admin]

Has anyone opened a Support Request on vDR? If not, please do so, as it needs to de documented and forwarded to product support team to review. Thanks.

VMware-GSS,

VMware Communities User Moderator

[shcrgeek]

Has anyone opened a Support Request on vDR? If not, please do so, as it needs to de documented and forwarded to product support team to review. Thanks.

VMware-GSS,

VMware Communities User Moderator

I have, on Friday. Provided ~1gb worth of logs. I'm having snapshot issues, problems with incrementals vs. fulls, and speed issues.

[Mirko_Huth]

I have exactly the same issues that you and pharmer have.... :smileysad:

[admin]

Some thoughts - if possible, the suggestion to open up an SR is the right one since it will help the team determine root cause

1) Make sure tools is updated on VMs - especially for Windows VMs since we leverage VSS to quiese the VMs. You mention that you are using VDR to protect the OS partitions, what are the OS?

2) There is no needs to scan the virtual disks for HW7 VMs since the vmkernel provides a changed block to VDR, while for HW4 VMs, VDR needs to scan the virtual disks at every backup to determnine changed blocks. Do you have both VM HW revs in your environment? Do you see a performance difference?

3) Re: destination disk, performance of vmdks and RDM has observed to be better than CIFS. Some dedupe store guidelines are posted here

http://viops.vmware.com/home/docs/DOC-1551

4) You can override the backup window by selecting the job, right click and choosing bring into compliance. This should begin backups of VMs that did not fit in window. Is you backup windows really 10pm-6pm or is it 10pm-6am?

5) Snapshots are executed by the vmkernel - VDR sends a call to execute one. So, still a VMware problem but the root cause is still with the snapshot engine.

6) I have observed random/odd issues with VDR in the communities where a reboot or a redeploy resolves the problem. It be worth trying both if your issues continue - note that the state is not lost since it is kept in the dedupe store. So, once you have redeployed appliance, reattaching the dedupe store/destination disk will bring the state back (jobs, logs, restore points)

[jarsenea]

You should note that my infrastructure is completely new since vSphere. We did not have any ESX servers (or VMs) until vSphere was released (this was by design). All our VMs are P2V's...

1) We're using Windows 2003 R2 (and a few Linux CentOS/Ubuntu VMs). VMware Tools is the one that comes with ESX4

2) We only have HW7 unless of course the VDR appliance uses HW4..

3) I guess I could use a LUN.. If there's a significant performance hit using CIFS that is. I'd prefer to use CIFS as I'm dumping the Store onto Tape from a Backup Proxy server.

4) My backup job is between 10pm and 6am. My mistake. I know I can bring things into compliance, but there definitely seems to be some problem doing this. I had a "copying <VM-NAME>..." task that ran for 8 hours (no reason this should happen) - the VM was only 20gb in size and it wasn't the first backup. I had to reboot the VDR (I've done this on numerous occasions, it's getting frustrating).

5) I had this at the start, but less and less.. This isn't so much a concern now (for snapshots) as it seems to be working (just thought I'd mention it)

6) Redeploying the VDR is a PITA for our environment - the reason being is that our network assigns externally-accessible IP addresses to each machine that connects (we don't have a company-wide firewall), hence I need to configure iptables/shorewall on each of my appliances. It gets frustrating having to reconfigure this everytime.

FYI.. I was unable to mount a CIFS share unless I did a :

yum install samba-client (which also installs samba-common). Once this was done, CIFS would mount.

[admin]

I don't want to troubleshoot the problem in the forums - and it seems that you have done a lot of work to get it to this point. Probably best if you were to open up an SR and really see what is going on. The fact that this is brand new vSphere environment is a good thing since this is the VDR sweet spot. However, troubling that you are running into a variety of issues - I can see one issue but as many as you have seen is troubling. Thus why I am somewhat puzzled.

Having said that

1) It would be interesting to know if the performance issues are related to the CIFs share at all. A test of backing up to an RDM or VMDK should provide us this data point.

2) In terms redeploying - again, some of the issues that you are seeing are fairly random (snapshots failing, copying an unchanged OS disk for 8 hours. Again, other contributors to the forum have resolved random issues by redeploying. If it is a PITA, then probably best to get with VMware support and see what they suggest - it may come down to this but maybe it can be postpone to the absolute end. One though around redeploy would be

1) Power off original VDR

2) Deploy new VDR appliance

3 ) Assign IP address of original VDR to new VDR appliance

4) Attach original destination disk to new VDR appliance

Will this still require a reconfiguration of your network (just familiar enough with your constraints)

Another thought that came to mind is the fact that you are using separate arrays for your source vmdk and the destination for the backups. This means that all the data blocks are travelling over the LAN - not sure what the network bandwidth is and what the backup traffic is contending with (i.e are the VDR backups running at the same time as the TSM backups over the same network?).

Are the source vmdks (OS disks) on shared storage?

[pharmer]

Support request opened re: VDR timezone issues: SR #1425741381

[VMSpotlight]

VDR is definatly a step in the right direction. The problem is it doesn't really cut it as an Enterprise solution.

Have you checked out esXpress at all? It's the same concept as VDR (VMs backing up VMs) except it's been around a lot longer and is much more 'polished'

______________________________________________________________________________________________________________________

Community Rep for PHD Virtual Technologies Inc

esXpress radically alters the notion of how to protect data in virtual infrastructures in one simple way: we use the virtual infrastructure to back itself up!

______________________________________________________________________________________________________________________

[Oletho]

But esXpress does not support vSphere, right?

Ole Thomsen

[certtrainer]

I believe the new version does- going to test this week.

[petedr]

Ole,

You are correct that the current version of esXpress does not support vSphere.

However the esXpress 3.5-9 which will be released this week will support vSphere hosts.

Pete@esXpress

www.phdvirtual.com, makers of esXpress

[petedr]

A follow up, the esXpress 3.5-9 release which supports vSphere is now available.

Pete@esXpress

www.phdvirtual.com, makers of esXpress

[jarsenea]

Let me add this issue to the mix:

"Trouble reading from the destination volume, error -2241 ( Destination index invalid/damaged)."

I can no longer use VDR with this CIFS share (unless of course I nuke the current backups, which kind of defeats the purpose of VDR!). I've tried restarting the appliance, redeploying via the OVF template..

This really should never have gotten out of beta.

[RParker]

This really should never have gotten out of beta.

Yet ANOTHER reason they need to have a PRIVATE beta among those of us on THIS forum that use ESX DAILY. Everytime I post this no one at VM Ware acknowledges this, YET we continue to have products released which CLEARLY are not ready and to the point 'never should have been released from BETA' is valid.

I keep posting and harping about needing some type of BETA program for even vExperts or a SPECIAL closed BETA so WE can test it ( WE being the people that umm.. actually USE the product, gee what a novel concept). Because apparently whoever BETA tests at VM Ware (other than being taken out and beaten with a wet noodle and fingers removed so they can NEVER touch software again) don't actually USE the product, they just assume that since the appliance runs ... well gee it looks fine! Yeah release it!

NOT!

3 times now this has caused serious undermining issues, the first being that debacle last year when 3.5 FIRST came out .. the 2 weeks time out where VM's wouldn't start... That should have been a clue to VM Ware then that the END users should actually have a vested interest in making sure the product.. uh.. let me .. oh yeah.. WORK?!?!? Interesting.

But no, VM Ware and other beta testers want to wave the 'NDA' flag, well phooey!

You want an actual WORKING product or not, VMWare? You want to keep it among 15 people in California who couldn't BETA test if their life depended on it.. that would be secondary to actually letting people ensure that its ready.

Gee I am not an Enterprise CEO, but that doesn't make sense to me.

BETA, it should be given to those competent enough to ACTUALLY test it THOROUGHLY before giving it a green light. This is not being done at the present.

Time and time again, we deal with inept, lackluster, faulty software. Is this an enterprise class product or not?

Oh it is?!??! Well why doesn't someone TREAT it as such?

[ruddg]

Have to agree here. vDR is not even a beta quality product... unbelievably poor.

I am seeing this same problem on a CIFS share. Tried 1.0.0 and 1.0.1. Looks like I'll have to nuke the whole thing... don't have time to waste on SRs for crud like this!

[Stevester]

Wow!!!!!!!!!!!!!! VDR is that bad huh. Well i guess i will have to wait and continue to use the GhettoVCB.sh script. In particular, I have a 120 GB physical machine which i am eventually going to convert to a 120GB VM. I would really NOT like to continue performing full backups of that machine every night via ghettoVCB.sh. I need a data deduplication system and we are entitled to VDR. Guess i will have to wait,lol!!!!!!!!!!!!!!!!!

Stevester

[TAMW]

Afwezig,

Zoals de meeste van jullie weten ben ik op dit moment vrij, ik ben weer aanwezig op maandag 7 september. Voor dringende zaken kunt u bij Erik de Wit (tel: [phone]) of Jan-Willem van den Hoek (tel: [phone])terecht.

Met vriendelijke groet,

Gertjan Pruim

[Stevester]

Can this be converted to English, i cant understand. LOL!!!!!!!!!!!!!!!

[KBuchanan]

Well - for me, it is OFFICIAL. I am not going to run the VDR until I hear that it is running without any problems. After much effort, I am throwing the towel in the ring. Maybe the next release will be more impressive - and actually work reliably. Even though I have heard VMware say they have customers running it without problems - I JUST DON'T BELIEVE IT.

But - to anyone that is interested, here is a script that is set to run as a cron job on each of the ESX hosts in farm. I have to balance the times that they run - but, at least they running SUCCESSFULLY. I only have 6 ESX servers, so the overhead with managing when each host begins their backup is NOTHING compared to the overhead of trying to get VDR to work.

It isn't elegant - there isn't any notification if there is a failure - BUT HEY...NEITHER DOES VDR!

This is as simple as it can get. I have seen other elaborate backup scripts...but (for me) they are overkill. I have this set to save 3 copies - but you can modify it (just read through the code). The only thing you shoud really have to change, is the backup path. I am using my VDR datastore (/vmfs/volumes/ESXDataRecStore)...just change this to your backup destination path.

Good luck to everyone! We'll meet again when the next version of VDR is released!

This is the script below:

1. This script will enumerate the guests running on this

2. host, snap it, clone it, and remove the snaps

3. This script will also keep 3 versions of the backups

for VMXFILE in $(/usr/bin/vmware-cmd -l)

do

FLDRNAME=`dirname $VMXFILE`;

VMXCFG=`basename $VMXFILE`;

GUESTNAME=`basename $VMXFILE .vmx`

BKUPDIR=/vmfs/volumes/ESXDataRecStore/$GUESTNAME

http:// -d $BKUPDIR.1 || ( mkdir $BKUPDIR.1 && touch $BKUPDIR.1/emptydir.txt; )

http:// -d $BKUPDIR.2 || ( mkdir $BKUPDIR.2 && touch $BKUPDIR.2/emptydir.txt; )

http:// -d $BKUPDIR.3 || ( mkdir $BKUPDIR.3 && touch $BKUPDIR.2/emptydir.txt; )

rm -f $BKUPDIR.3/*

mv $BKUPDIR.2/* $BKUPDIR.3/

mv $BKUPDIR.1/* $BKUPDIR.2/

cp $VMXFILE $BKUPDIR.1

/usr/bin/vmware-cmd $VMXFILE createsnapshot BackupSnap BackupSnap 1 0;

for VMDKFILE in $(find $FLDRNAME -type f -name "$GUESTNAME.vmdk")

do

VMDKFILENAME=`basename $VMDKFILE`

/usr/sbin/vmkfstools -i $VMDKFILE -d thin $BKUPDIR.1/$VMDKFILENAME

done

for VMDKFILE in $(find $FLDRNAME -type f -name "$GUESTNAME"_"?.vmdk")

do

VMDKFILENAME=`basename $VMDKFILE`

/usr/sbin/vmkfstools -i $VMDKFILE -d thin $BKUPDIR.1/$VMDKFILENAME

done

/usr/bin/vmware-cmd $VMXFILE removesnapshots

done

[XavierE]

I give up with VDR as well. I have been testing well enough and I've been losing time.

I will keep using the script I wrote in the meantime, it's simple, nothing complex and has been working fine for me.

The script is attached if anyone'd like to use it. Here are some notes:

• The script should run on the VCB proxy server.

• There's a part to fill out variables to suit it to your environment

• It sends notifications when it's done and attaches the log.

• It also notifies about general failure backups.

[Stevester]

Does anyone know if an open source data deduplication or freeware solution has been developed or is in the making.

Just curious

Stevester

[FBlanchet]

Take a look at Windows Storage Server 2008 (WSS), witch is an embedded version of windows 2008, that you will use ase an ISCSI target, it runs data dedup. I have the ISO, named "en_windows_storage_server_2008_embedded_basic_standard_enterprise_workgroup_dvd_x64_x15-49574.iso" contact me by email if you want it...

[Stevester]

Can this do deduplication at the VM snapshot level as VDR does?

[XavierE]

You may have seen this one already

http://vmblog.com/archive/2007/01/17/open-source-backup-and-data-de-duplication-virtual-appliance.aspx

[Stevester]

I saw this and I assume this is only a virtual appliance that does file level deduplication. Can it do deduplication on VM snapshots as VDR does?

Stevester

[chriswaltham]

I'm having lots of issues, but they're different issues to what most of you seem to be having. They are frustrating as hell, though!

We have one vSphere deployment; three clusters with a total of 13 hosts & 170 or so VMs of Windows & Linux flavors. (We have 2 VDR VMs because of the 100 VMs-per-VDR-VM limit.) All our data lives on NetApp filers; the ESX hosts run the VMs via NFS. The VDR VMs are using virtual disks that are actually VMDKs attached to the ESX hosts via NFS. (So, no CIFS involved.)

My biggest problem is simply that that backups don't run, even when they're scheduled. And, usually after a day of being up, the VDR VM is inaccessible to the VDR plugin -- that is, when I try and connect to the VDR VM with the plugin, it says that the host couldn't be contacted. I can still ping & SSH to the VM, but I have to reboot it for the VDR plugin to be able to connect to it.

I wouldn't have these concerns if this was beta software, but seeing as it's production it is pretty tough to deal with. I already have a SR open (1449882361) but we haven't identified anything yet...

Chris

[Elgreco]

The same error here...destioation volume error 2241

vdr is still full of bugs and i wonder why they put it in the production ...

[chrisaug]

I just want to say that so far we are testing with VDR 1.0.0.153 (Build 164675) and so far (crossing my fingers) we haven't had any issues. Backups are running in the scheduled time frames. Backups are completing succesfully. My only errors occur when I try to backup a VM that is powered off (duh).

I left the defaults of the OVF template for the VM (2GB of RAM, 2 vCPU). The only thing I changed was from DHCP to Static IP. Storage is a NetApp FAS 2050. The backups go to a CIFS share on the SAN. Haven't had any issues yet. I'll keep updating this as we go along with testing, but I have to say... things are looking promising (maybe I'm just too optimistic).

And just as an aside - I am testing VEEAM Backup as well and I must say although it is much more granular in its backup and restore processes it takes almost 2 to sometimes 4 times as long to complete a backup of 8 test servers

[Stevester]

Chrisaug: How are your backups doing now. Have you had any issues with VDR?

[XavierE]

I did have issues. I noticed that if you have backups already taken and you change the structure tree in the vsphere (folders, datacenters, clusters etc) you cannot restore your backups anymore.

[XavierE]

This VDR is a great attempt but clearly not ready for production yet. VMware should speed up on releasing a stable and very well tested VDR. (they are supposed to release a new version by the end of june which supports file-level restore)I do also have several issues like many others in this post. Briefly I will list them briefly

1. Operations take too long to perform (regardlesss your destination location)

2. Backups fail without helpful information

3. Backups do not start at the backup intervals

4. VDR crashes during backup/restore

5. snapshot failures

[admin]

I appreciate the comments and we take them to heart. When we GA a product, it has done through customer and internal validation that matches our exit criteria for GA. However, we know that it will not be a perfect product and that "real world" testing by customers help us make a better product. I am not sure if you submitted any SR for the issues that you outlined, but that will provide more data about your environment/testing. This then drives changes in our internal QA environment, which we hope will close that gap in our QA test matrix. Some comments

1) The file restore client is available today for Windows (experimental support). Sorry I don't have link handy, but it is in one of the forum posts

2) We have seen some snapshot and connectivity issues when name resolution does not occur. Most customers have been to work around this by adding the hostnames in the \etc\hosts files on the VDR appliance.

3) You are correct that backups do not start exactly at the opening of the backup window. However, unless there is a VDR process running that "locks" the dedupe store (integrity check for example), the backups should start soon after the backup window. I usually look for consistency of time stamps of the restore points - (backed up roughly the same time each day)

4) When you saw "VDR crashes" - what specifically is happening (error messages, state of the VDR virtual appliance), etc?

5) When the backups fail, there should be an error message (yes, I agree that some may not be customer deciperable) - can you share what these error messages were?

[XavierE]

Answer to your items:

3) Backup window is between 5pm and 8am. after I created the backup job it ran right away and that's the only backup I have after 3 days. The status shows as waiting/idle and there's nothing else locking it. I have not manually force it because I want to test that the backup window actually works.

4) In a separate test env. The VDR would crash (reboot) in the middle of a backup. The logs would just show errors saying the task terminated unexpectedly possibly due to a power failure or system crash.

5) Backup failures show different messages like

• trouble writing to destination volume, error -22,

• the task terminated unexpectedly possibly due to a power failure or system crash

• can't access backup set, execution errors

I have tested the destinations (network shared & vmdk in SAN) and they seem fine no problems with the I/O.

Name resolution is not a problem

[RParker]

I appreciate the comments and we take them to heart.

I'm sorry but that's not good enough. I don't think YOU have any authority, and given the fact the VM Ware products over the past year have been LESS then stellar at release, or like now obviously BETA material, certainly NOT GA release, that doesn't seem to be the case.

Maybe you THINK they take it to heart, but actions speak louder than words. I see no changes in the way VM Ware releases their product, that is issue.

Why can't we being the people who buy, use and manage this software test it BEFORE you check it off the list to be release? That's all we are asking.

Take that heart and get back to me.

[XavierE]

VDR used: Version: 1.0.1 | 2009/07/09 | Build: 176771

I am still unhappy with VDR. I just deployed it, did a test backup (passed this point at least) now I am trying a restore to benchmark the restore time and I get this error

Trouble writing files error -3956 (operation failed)

Tried different ways, same result every time.

Destination is a CIF folder.

We may get a decent VDR by the time they release version 2.0.0 some time next year perhaps HA!

[wildcattdw]

I'd be interested to see what other people have done for troubleshooting. Personally, I started with only CIFS destinations, but after countless issues, tried a local VMDK connected via FC, which still yielded two resotre points with errors in less than four days of operation.

Today I bumped the memory on one of my vDR appliances to 4GB, and with an uptime of only 24 minutes, 'top' is reporting 3.3 Gb used, less than 600k free memory... and this appliance is running a "reclaim" right now, no backups have occurred since booting it. That doesn't seem right to me.

T

[jarsenea]

VDR is hosed, plain and simple.

I'm embarassed to have to go to my superiors in order to find money to buy vRanger or esXpress. VDR was touted as a solution for small environments (we only have 25 VMs) and now I'm forced to spend MORE money to buy a competing product.

Talk about false advertisements.

(And yes, I have SRs open but they all seem to come back as "confirmed as bug").. QA really blew it on this one. VDR isn't even beta quality, it's alpha quality.. hell, maybe not even alpha quality since it doesn't work most of the time.

[ithotlinedothu]

We are testing VDR trial version for a month.

Very, very poor performance. We tried find bottleneck problem our system, but wasnt found.

I was think problems caused our cheap iscsi, nas solutions not vmware HCL hardwares.

But reading this forum i think a same problem with HCL hardware.

Yeah yeah we are known if you plan backup solution, use other backup method, other software.

VDR is not ready for production use yet.

Just vmware why not say VDR is a beta, alpha software?

[Mirko_Huth]

I fully agree.

To bad that VDR was the only reason for me to upgrade to vSphere.

Now I can't use it and have new problems cause by vSphere itself (iSCSI performance, connection issues with SAN storage).

Currently thinking about going back to VI3.5. ...

[KBuchanan]

There are lots of issues, and I am running a script to backup the VMs. It works, but I would rather see VDR working reliably. I have highend storage for destination...but it seems to be a bigger problem than just the storage.

[Mirko_Huth]

I stopped wasting my time with it. It's a fact that the system is unreliable and I can't recommend the usage in a productive environment.

I assume those people having no problems have only few machines to backup. Problems seem to get worse when the datastore grows.

Also I don't like the idea to have everything in a single datastore that can get corrupted without the option to get the vm out of it except by using VDR. And the fact that there is no reporting/alerting is disappointing.

I'll continue to use VCB. It's reliable and simple to use. If Vmware would just wrap a gui around it it would be perfect and there would be no need for something else.

Deduplication and several versions of a VM are nice features. But knowing to have a reliable and restoreable backup is more important to me.

Mirko

[KBuchanan]

Well said. Maybe VDR 2.0 will be a more reliable release.

[KBuchanan]

I opened a SR every time I can't get the backups to run. If they (VMware) aren't aware of our issues, they can't fix them.

BTW - my destination is on (14) 72G 15k FC dedicated FC disks in an EMC 3-20 cabinet.

[JamesSykes]

Having all of these issues myself. VDR is defintely not ready for a small business let alone enterprise.

What is most annoying though is the constant WAITING. I get hardly any feedback as to what is going on.

Failing snapshots, random terminations, slow backing up etc..etc..etc...etc..

Really don't see how it ended up in 4.0 in this state. This isn't even beta level.

[MathiasC]

Hi!

I have a different set of problems...

When I tried to do a rehersal restore, there's no hosts showing up in the list, and to be able to select another destination for the restore, I had to move the VM to another cluster. If I didn't, there's only ESX host local disks showing up in the destination list.

If I select another host for destination, the restore-VM disappears...

Anyone else seen this behavior?

/Mathias

[TAMW]

There is a VDR update!

Latest Released Version: 1.0.1 | 07/09/09 | 176771

The v1.0.1 release addresses the following

Large Temporary Files Removed as Expected

Data Recovery modifies virtual machines' vmdk files' settings so a snapshot can be created for backup purposes. In the past, after the backup has been created, the vmdk file's settings was sometimes left configured for snapshots even after the backup was complete. This led to these virtual machines being left in snapshot mode while accumulating snapshots that were undetected by vSphere Client. This process has been redesigned so that these temporary files are no longer be left behind. In previous versions of Data Recovery, this issue can be resolved by following the process described in the knowledge base article titled "Delete ddb.delete entries and snapshots left behind by Vmware Data Recovery".

Backups Can Be Completed While Integrity Checks Are Running

Data Recovery can complete backup operations at the same time that an integrity check is running. In the past, when an integrity check was running, backups could not be completed.

Improved Integrity Check Backup Speed

Integrity check has been optimized for faster performance. In the past, comparable integrity checks took longer to complete.

Improved VMotion Licensing Support

Virtual machines can be moved between hosts using VMotion without producing licensing issues. In the past, if a virtual machine was moved between hosts using VMotion, licensing checks sometimes produced errors.

Reduced Data Recovery Backup Appliance Shutdown Time

The Data Recovery Backup appliance now shuts down more quickly than it did before. In the past, the appliance often took 15 minutes to shutdown.

Improved Support for Different Time Zones

In the past, Data Recovery did not consistently handle time zones with positive offsets relative to GMT. For example, Data Recovery could encounter issues with data associated with the Paris time zone, which has an offset of +1, whereas data associated with the New York time zone, which has an offset of -5 was handled as expected. These issues no longer occur.

Data Recovery Supported with Essentials Plus Licenses

Data Recovery is included in Essentials Plus licenses. In the past, using Data Recovery with Essentials Plus licenses failed. Backup jobs created with Essentials Plus licenses failed with the error "License not available to perform operation. Feature hotplug not licensed..."

Integrity Check Optimized to Run During Idle Times

Before running regularly scheduled integrity checks, the Backup Appliance determines if the current time is during a backup window. If the current time is not during a backup window, the integrity check runs. If the current time is during a backup window, the backup appliance checks the backup schedule to determine if there will be a time in the next 24 hours that will not be during a backup window. If there is a time in the next 24 hours that is not during a backup window, the Backup Appliance waits for that time. If there is no time that is not during a backup window in the next 24 hours, the Backup Appliance completes the integrity check

[certtrainer]

Wow-

I wish I had the same issues as what was reported above. I can't even get as far as having a slowdown. Initially I was getting 3902 errors (file access error) when trying to perform an initial backup. Of course, there doesn't seem to be any information on that error.

I couldn't log in as root onto the appliance (it would just hang). So, I rebooted the appliance, then gave it a static IP. Even though it seems to be configured correctly, I can not open the tool in the vSphere Client. (Could not connect: The operation is not allowed on non-connected sockets).

Don't think I will bother putting in an SR- this product does not look reliable.

Just venting here- echoing what others feel about the product not being ready. I guess I'll look for a third-party product. Very disappointed in a feature that was one of the main reasons we went to ver 4.

[RParker]

Don't think I will bother putting in an SR- this product does not look reliable.

VDR 1.01 is better.. have you tried that, NOT the one that was release with ESX 4.0 this is an update . . . . .

[KBuchanan]

I am running v1.0.1 - and it is still buggy to some degree. I opened SR 1431443551 because I am getting integrity errors, some of the machines fail to backup, and I can't restore some of the images.

Well - for a backup product, VDR is failing miserably. I would like to hear that someone is actually successful with the backups and then understand why their experience with VDR isn't repeatable. There is a lot of "problems" circulating in the forum - which tells me VDR has some serious flaws for the masses!

[pkurnaev]

Hi everybody! I`ve got the same issue - VDR crashes while backing up VM with large virtual disks (40250250Gb).

Did anyone noticed relationship between VM's size and backup process reliability?

[mgaub]

Hi everybody! I have the same problems. I have VDR 1.0.1 running for a few days, backing up 21 VM's without any problem. As soon as I add a VM with large disks (24500100100100 GB) VDR crashes. After that, my destination volume is corrupt with error -2241, and the integrity check stucks at 33%. I had to reformat the volume to make it working again!

Marc

[Steve5]

Well, I can't even get a backup to run. I'm running 1.0.1.362 with a local share (why can't I browse locally?) and I've had to format my destination twice to pass the integrity check.

Immediately after starting a backup using Backup Now, the job fails and I get:

"Can't access Backup Set /my ip/my share, error -2246 (wrong Destination index found)

This is fresh install of VDR on a fresh install of VC. I've changed the local share username and password to alphanumerics with no change to the error.

[certtrainer]

As per my post above-

It still does not work correctly.

[KBuchanan]

Frustrating, isn't it? I have an SR opened (yesterday) - so, I am expecting a follow=up today. I'll post the outcome here. I am very anxious to know that this product "DOES INDEED" perform as VMWare states - but, so far, I can't say it can perform "as stated" with a high degree of reliability and repeatability.

Kevin

[RParker]

product "DOES INDEED" perform as VMWare states

Can you say with reasonable amount of certainty that your backups are SAN based and not network? I figured out I am getting a permissions error (vcbAPI) not sure why that is...

[KBuchanan]

RParker wrote: product "DOES INDEED" perform as VMWare states

Can you say with reasonable amount of certainty that your backups are SAN based and not network? I figured out I am getting a permissions error (vcbAPI) not sure why that is...

I'm not sure if you were directing your comment to me. But, I am backing up the VMs to SAN storage - and regrettably, it is SAN storage in the "same rack". I would prefer to have the VDR backing up to a NFS share in another data center. I tried to do that, and I kept getting failures.

Currently, I have 4 days of successful with VDR. I have been able to sucessfully restore each VM to a "re-named/re-directed VM". I had a SR opened regarding failures with backups (see this thread). I was getting a Incompatible device backing specified for device '1' error. The VM Support Tech admitted there was a bug. I posted the resolution and a workaround is in the link.

The largest VM I am backing up is around 60G. I don't have enough confidence in VDR to say it is reliable, and I have strong reservations that it can backup to a storage device across the network (ie, CIFS, NFS). I tried NFS, and kept getting numerous backup failures and integrity check errors. I am not getting any errors backing up to a SAN.

My advice: TEST IT, and it your test is sucessful, TEST IT AGAIN. Keep testing every few days to see if it can restore - and WATCH THOSE LOGS!

[FBlanchet]

Hi, I have exactly the sames issues. Not ready the deploy as a production envirronement backup solution. Will try 1.0.1 VDR

[RParker]

Will try 1.0.1 VDR

Yup back to Vizioncore

7/17/ - VDR 1.01 seems to be better, I can login now, and so far no snapshot snafus.. although 1 VM (with multiple previous snapshots gives an error, and it will not delete the snapshot). If I check the VM manually, it shows the snapshot as gone... Everything else is .. dare I say it.. adequate (not perfect)

Message was edited by: RParker

[RParker]

Having all of these issues myself. VDR is defintely not ready for a small business let alone enterprise.

Or even testing... I can't even TEST with it, it's so rediculous.

What is most annoying though is the constant WAITING. I get hardly any feedback as to what is going on.

Or from the engineers at VM Ware.

Failing snapshots, random terminations, slow backing up etc..etc..etc...etc..

Poor implementation, remedial sparse documentation, poor customization, limited options . . . . .

Really don't see how it ended up in 4.0 in this state. This isn't even beta level.

I second that! It's like pre-Alpha.

[06blade]

the only way I've managed to resolve the snapshot issues are to create another storage destination.

It works for a while.....before screwing up again.

[alexlie]

I'm using VDR 1.0.1.362 with two local destinations disks with ~2 TB total (via iSCSI)

I've notices the following issues:

1) If a vm has more than one disks with the same name (e.g. xyz.vmdk) on different datastores the restore tab only shows the first disk on the vm after the backup

2) In the navigation pane some machines show up with a nested second machine with the same name when expanded (it's not recursive - despite the +-sign the nested instance does not have a vmdk)

3) During the last backup job some vmdks vanished in the navigation pane - the are all back now

4) Got some errors: -3941 (create snapshot failed), "failed to create snapshot for xyz, error -1" - Don't know what to do.

5) Got "Destination "/SCSI-0:2/" will be locked until the restore point with errors are delete and integrity check succeeds." - Nice, what should I do? Which are the restore points with errors?

6) During the last automatic integrity check I've noticed a task without name and with start-time - what's that?

7) All in all backups took very long (>2 days for machines with ~200 GB disks)

I really like the concept of VDR but in the current state it is barely useable nor can I rely on it as our only backup solution.

[KBuchanan]

I would be surprised if VMware is actually reading this.

Kevin Buchanan

Chief Information Officer

Lexington Memorial Hospital

336-238-4286

[e-mail]

[alexlie]

I would be surprised if VMware is actually reading this.

In fact they do! I am pleasantly surprised.

[admin]

Alexie

A) You mentioned iSCSI as your destination disk? Can you confirm that the iSCSI array is listed in the VMware storage compatibilty list

http://www.vmware.com/resources/compatibility/search.php

B) Are the disks destination disk thick or thin provisioned? We have some recommendations on this here:

http://viops.vmware.com/home/docs/DOC-1551

We have seen a lot of issues with customers using non-HCL compliant storage. They fall under one of two categories

a. Low end NAS solutionsthat support CIFS/iSCSI/NFS and additional protocols.

b. Repurposed Windows 200x shares.

Obviously, how efficient these types of destination disks are will impact how well VDR performs. Our recommendation is to utilize HCL compliane storage for VDR destination disks.

On your questions

1) My guess is that this is a bug - thanks for finding it. However, is this something that is typical in your environment?

2) Which navigation pane? Main, Backup or Restore?

3) Which navigation pane? Main, Backup or Restore?

4) I know that this is passing the buck, but snapshots are executed by the vmkernel - VDR just makes an API call and reports back the result. Has a VDR snapshot (i.e subsequent backup) failed? I know of some customers that have said that rebooting the VDR appliance solves this problem, but we are not convinced that they are related.

5) In VDR/Configuration/Logs, scan to the last failed integrity check and you will see the restore points with errors. The quick workaround is to manually mark these restore points for deletion, though obviously, we are concerned about how these restore points got into this error state to begin with. Do you see any clues in the logs around errors for these VMs/restore points?

6) Don't know - if you see it again, can you get a screen shot?

7) Obviously, the backup duration will be dependent on amount of data to be copied and how many of the vSphere optimization is being utilized?

How many VMs were being backed up per backup job?

I assume these VMs had been backed up before and there are restore points in the dedupe store?

Are the VMs HW 4 or HW7 (HW is better since we do not have to scan to the disks to determine changed blocks?

I assume that the logs state that the snapshots are being hot-added to the VDR appliance (as opposed data being copied over the network)? Obviously, copying over the network will be much slower that performing hot-adds of the snapshots

[ehinkle]

Point number one is a known issue with esx3.5 and creating snapshots on a vm with virtualdisk having the same name on different data stores you will need to rename the virtual disk.

KB 5096672 -http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=5096672&sliceId=1&docTypeID=DT_KB_1_1&dialogID=32222594&stateId=0%200%2028722776

[alexlie]

Thanks for taking care of my post. Here are my answers:

A) The VDR destinations are on an EMC Celerra NS502, the primary datastores are on two EMC CX3 series Clariions. The LUNs are connected via Qlogic iSCSI HBAs on the ESX 4 server. All are covered by the VMware HCL.

B) Disks are thick (also on the Celerra side)

... so our storage should be fine. I agree with your concerns about "low-end" storage systems... that's why I don't use them - I do not want to have to backup my backup :smileywink:

1) It's not typical... I ran out of storage for some machines and had to use different datastores. I ordered a new disk shelf last week which should solve this issue (after renaming the vmdk's - thanks to ehinkle for pointing this out)

2+3) The big navigation pane on the left side which is always visible. See screenshot.

4) I will look into this.

5) Found the problematic restore point. Also found that the corresponding backup quit with "Trouble writing to destination volume, error -107 (out of application memory)" - Should I try to increase the memory for the appliance?

6) If the mysterious task shows up again I will make a screenshot.

7) Yes the snapshots are hot-added. We currently upgrading our vms to HW7. I hope the performance will improve. I have two jobs with each ~15 machines scheduled. As described in the manual vdr runs 8 tasks concurrently.

Again, thanks for your reply.

[XavierE]

I've been testing VDR since the first version was released.

Marketing and webex presentations looked great but a very different story in a production environment.

Numerous problems like many people in this post and it's have been just a waste of time/money

If VMware is actually checking this post I think they should RECALL this product from a RTM, apologize at least and either drop it from their products list or work on it until it's fully tested and decently working.

Who knows they could even pursue on buying a competing company like vizion core etc, and rebrand their products (like they've done before and other companies do nowadays)

I am sticking with VCB in the meantime.

[KBuchanan]

VMware is listening and reading these forums. Believe it or not, I received a call from the VDR product mgr.

In my opinion, we may have to wait for a few more updates for it to work for the masses. I have had problems and several SRs...still getting mixed results.

I suggest everyone to test VDR, but keep another backup solution in place.

[adisarro]

We're testing VDR while still running VCB for good measure(and as the easiest way to get to tape for offsite). I have not yet seen any of the issues above. I have only been using it for 2 days though so i'll be sure to post back here again once i have more testing done. I just felt the need to post an everythings-fine-for-me post so people reading don't think that it's 100% crap.

Backing up about 25 vms on the two vsphere 4 hosts we have(still 4 3.5s we didn't upgrade yet)

400gigs worth of data(thin provisioned, not including free space)

Destination is a home built win2k3 box over cifs with 3 1tb drives in a raid 5

so far the 400gigs has been backed up for two days while using only 86gigs :smileyhappy:

[wildcattdw]

I was extremely excited about the inclusion of a backup solution... and when I started testing with it everything was roses. I was backing up 8 machines that were VERY identical; total of ~120 GB used space backing up to something like 35 GB after a week. Once I started rolling it out to a broader range of guests, things took a very quick turn for the worse. I currently have 4 appliances in different configs, different targets, and all are displaying issues.

Makes me sad panda.

[admin]

Worked fine for two weeks then blew up.

The destination is corrupt and all the backups are useless, nothing I tried (not that you can do much) would not get it going. I wiped the datastore and added a blank one and the appliance simply would not start backup jobs. I'm now deleting everything related to it, I wanted to try the latest release but once again the vmware store is down for maintainance (is it just me or do they take that down for 24hours every two weeks or so?)

Wish this worked, I've been extolling the benefits to clients, but there's no way I would install it in production.

[Stevester]

Are there any open source alternatives to VDR that does data duplication?

Stevester

[KBuchanan]

Would you really trust open source sw? ...dont get me wrong, I just don't think that open source is the best option for protecting production systems - unless there is a an adequate support contract.

Use VCB. Or, if you want to have a free solution, search for vcbghetto backup script.

I am using a script I developed myself to perform backups. Until VDR is more reliable, I suggest you have a solution in place to protect your images until VDR proves itself for you.

[Stevester]

Red Hat Linux is open source, and in fact the ESX Server, is a Red Hat implementation. I do agree with you!!!!! Open source is a double edged sword, but less us not count it out totally!!!!!!!!!!!!!!!!! I already use ghettoVCB but i need a backup solution that has data deduplication built into it.

[KBuchanan]

Read what I said...I don't suggest open source, unless there is an adequate support agreement.

We user open source OS and software in our LIB apps. BUT we have support contracts as appropriate for the application

I'm an advocate for open source, but if there is a problem, you have to be able to contact the vendor for support.

[Stevester]

I read what you said, and once again, less not totally count out an open source solution. There is no contract with GhettoVCB and yet its used and very popular I might add.

[KBuchanan]

Fully agree with you.

Open source an community supported software has its place within a business IT infrastructure. I would be careful that someone not confuse free open source as always a viable solution.

Anyway...I am still running VDR and I am hoping that VMware is able to fix the problems reported in the community postings...but it is our responsibility to report the problems as an SR case so that they get an officially recorded statement of the problem. Until VMware releases a version that is reliable in my environment, I will continue to test and work with their tech support. The VDR features are incredibly outstanding...I just need to see it work consistently and reliably well in my environment.

[Stevester]

Let's all just give VDR a chance. You have to crawl before you can walk. Its gonna take time before it gets polished and consistent. I am sure the VMWare engineers are well aware of what needs to be done. We just need to give them a chance and be patient.

Take care everyone!!!!!

Stevester

[KBuchanan]

Absolutely!

Believe it or not, I got a call last week from the VMware Product Mgr for VDR. He is actively reading the community threads, and I believe the VDR product will ROCK - in the next major release. Most of the problems come from people using poor quality storage destinations - although, I have had numerous problems on high end FC destinations.

Anyway - I think it is important to test VDR and report problems as an SR case. Don't complain about VDR in the forums, unless you are opening an SR. Opening a SR will ultimately be beneficial to everyone because it helps them to better understand how VDR is being used in the various different configurations.

I'm just as quick to point out my frustrations with VDR b/c I think there some serious problems under the hood...which I why I have several open SR cases now.

I think that when another major release of VDR is available, it will be more reliable. I just hope that it doesn't lose too much credibility before then.

[Mirko_Huth]

Not only the product lost it's credibility vmware also did in my opinion. VDR is not a beta product. That's why customers can expect it to work as is.

I'm really disappointed that VMware is not feeling responsible and is moving it's quality control to the customers site!

It's not ready yet and VMware product mgmt seemed to ignore the developement progress just to lauch it together with vSphere.

And blaming the customers Hardware is not the right way to handle insufficient testing done by vmware.

By the way, has anyone seen an increase of iSCSI performance by factor 10 in v4phere as promised? I doubt...

Just my opinion...

[KBuchanan]

I don't recall seeing any promise to increase iSCSI performance, but the performance doesn't appear any different in 4.0 vs 3.5.

As for VDR...we should treat this as a 1.0 product (well, I suppose that won't be too hard) and we should test it and report the problems to VMware. And for now, don't rely on it as a backup solution unless we see (collectively) that it is working for the masses.

VDR promises to be awesome - we just need to give it every opportunity.

[morciod]

This message is for Adisarro.

Are you running vdr from remote site across the wan to a datacenter network share? we are testing vdr against local vm's and have not had many issues. deploying to remote site and sending data back across the wan cannot be tested in our environment yet. have you any experiences?

[vAdmin]

so, in this case a file level restore is only supported for Windows VM ?

It would be great to have VDR as a working production backup solution.

Kind Regards,

AWT

[jarsenea]

I realize now that although VDR has improved since I originally posted this message, it still has a long way to go. I'm often getting failures backing up (even with 1.0.2). Simple fact of the matter is that (gasp) this software doesn't allow a sysadmin to do much in order to try to rectify any problems. For instance, I can't manually do a reclaim in order to delete corrupted backups - speaking of which, I get a lot of corrupted backups these days (Heck, got into the office this morning and 70% of my backups were corrupted - HOW IS THAT ACCEPTABLE?)

I guess you get what you pay for.. Wait a sec, I PAID A LOT FOR THIS.

[vAdmin]

Yes , you are not alone, I also feel the same anger with you.

Initially I was using vSphere Essential and just for the sake of getting a backup from this VDR Appliance, I paid nearly twice of the initial cost to get vSphere Essential 4.

Kind Regards,

AWT

[abbasi]

Did VMware even test this product before rolling it out? I have 300 GB of data backed up everything went corrupt

10/23/200 12:58:36 PM: Executing Recatalog

10/23/200 12:58:36 PM: To Backup Set /SCSI-0:1/...

10/23/200 12:58:37 PM: Trouble reading from the destination volume, error --241 (Destination index invalid/damaged)

10/23/200 12:58:37 PM: Backup Set "/SCSI-0:1" will be locked until the restore point with errors are deleted and integrity check succeeds

10/23/200 12:58:37 PM: Task incomplete

10/23/200 12:58:37 PM: Duration 00:00:01

Is anyone from VMware reading these forum postings and hearing our grief.

[KBuchanan]

Abbasi:

I can attest that VMware is reading these posts. Not long after I "ranted" about the reliability, I received a call from the VMware VDR Product Manager to discuss the issues I was experiencing. At a minimum, it is comforting that he took time to call and speak with me - however, the consolation isn't enough. I have to see the product (VDR) is working reliably for EVERYONE.

We have a responsibility to continue posting our frustration. They are reading/watching these forums. It seems fruitless, but I faith they are working to correct the deficiencies - unfortunately, it just isn't quick enough to meet my timeline.

Kevin

[XavierE]

I agree. VMware may be reading this thread, however time is money and they already got the money from us and what do we get? just wasted time. We -customers- have no other option but wait until the product is ready, in the meantime how do we explain to upper levels that the cost for this product was not an investment but an expense since the clock keeps ticking and in time being we cannot use because it's not reliable.

[abbasi]

I would encourage everybody that if you have paid support to file a SR as they will more likely take it seriously if there are a large number of SR's related to the product. I filled a SR for my issue but have yet to hear back and it is close to the 5 hour mark and we have Gold Support

I told my upper management I had a solution for backing up our VM's and it has gone down. No I have to go before them and tell them it has failed which reflect badly on me and I have to ask for $$$$ for one of the comercial products

I think this is fustrating b/c usually VMware puts out such great stuff and then this product breaks (and takes down all of your backup data) at the drop of a hat.

It is almost like nobody even tested it out

[vAdmin]

Yes very true, I'm stuck in this backup process testing, since all of my Server is ESXi, there is a 3rd party product that is very great but the ESXi support was taken by VMWare :-| this is sounds monopolistic ?

Kind Regards,

AWT

[KBuchanan]

Their restriction of ESXi is unethical to say the least.

Kevin Buchanan

Chief Information Officer

Lexington Memorial Hospital

Lexington, NC 27293

Ofc: 336-238-4286

Fax: [phone]

[KBuchanan]

I'm in the same boat...but I didn't tell my CEO we can't back up the images!!

We are using a script to backup the images to a NFS datastore hosted on Openfiler.

It is a "homegrown script" I wrote that enumerates all the guests, snaps them, and creates a backup image. Plus, it cycles the images everyday. It isn't as elegant as the ghetto script, but is works great.

I've said it before, no one should rely on a backup without FULLY testing it.

Good luck!!

Kevin Buchanan

Chief Information Officer

Lexington Memorial Hospital

Lexington, NC 27293

Ofc: 336-238-4286

Fax: [phone]

[KBuchanan]

You are right...we are left holding the bag. Maybe VMware will get their act together and get it fixed.

Kevin Buchanan

Chief Information Officer

Lexington Memorial Hospital

Lexington, NC 27293

Ofc: 336-238-4286

Fax: [phone]

[vAdmin]

Thanks for the reply Mr. Buchanan, in this case you're in the same boat as those who dissapointed with VDR appliance.

Can't wait until VDR 2.0 is released .. >_< seems so long..

Kind Regards,

AWT

[abbasi]

My Integrity check has been going on for 2 days now and is only at 44%, it only moved 5% in the past 24 hours I can't do any back-ups or restores while this is going on. At this rate I can only do monthly back-ups.

I opened a SR and was told this is normal and to just wait. Normal??? How can this be normal? The product is basiclly useless and should not have been released instead of using us as beta testers

Can the VDR Product Manager please come on and give some answers?

[Stevester]

Wow!!!!!! For a company like VMware that is embarrassing.

[KBuchanan]

The shame is that aren't recalling it.

Kevin Buchanan

Chief Information Officer

Lexington Memorial Hospital

Lexington, NC 27293

Ofc: 336-238-4286

Fax: [phone]

[vAdmin]

To All,

When you guys got a reply back from the SR or any words about new VDR patch, can you please share it here :-o) I just couldn't believe that this product is a paid commercial not a freeware :-| and yet still buggy as if it is in RTM / alpha version.

If still no words from VMWare, I'd better of go with 3rd party backup software with Dedupe option which can support ESXi

Thanks.

Kind Regards,

AWT

[chriswaltham]

Another issue I seem to have somewhat regularly is that, when trying to run a job, nothing happens. I mean nothing at all; no snapshots get created or anything. Of course, rebooting the VDR VM usually fixes this... usually.

[paulland]

I've had a SR open for well over a month. VDR worked (as far as I know) OK for a couple of weeks, then started throwing errors, and now it just sits there acting stupid. I've sent over a gig of logs to support and spent hours playing "Try this and see what happens".

I heard back from support a week or so ago. There's a patch coming for VDR to resolve these problems.

They couldn't say when it was coming, other than "soon". But rumor has it there's a big patch coming for ESX4, so maybe it'll be in there.

I've moved on. PHDvirtual's esXpress and its dedup feature is doing the job (and is much more flexible about where the backups live).

[abbasi]

I have had a SR open for 2 weeks now. The tech said they are aware of the many issues and a release will be coming out late Dec. He also mentioned there would be a service pack upgrade coming for ESX4 as well in that same time frame.

If you have not deployed VDR my advice is DO NOT USE IN A PRODUCTION ENVIROMENT

My first issue was that the integrity check was taking days to run despite the fact we had only made 3 days of backup. I posted the solution to that in another thread. As soon as I got that resolved a new problem started right away, the back-ups jobs were running when they were not supposed to in the middle of the day. I had a VMware tech look at and he didn't have a solution other than to wait it out as a Integrity check was going on at the same time. So after waiting several more days the 5 VM's are now stuck at various points in their back-up. I can't stop them and the Snapshots are now growing, they have been stuck for 7 days. I have had the same tech working on all these issues and I have been in contact with him everyday and honestly it doesn't seem like he has a lot of answers.

Also whenever I reboot the VDR server it doesn't remount the VDR datastore. I also just saw a thread where VDR deleted someones VMDK files, it is a known bug

VDR should have been released as 'Experimental'

At this point VMware should recall this product

[KBuchanan]

ABBASI:

I have already posted (several times) about:

- NO ONE SHOULD USE THIS IN PRODUCTION

- THE VDR PRODUCT IS FAULTY AND SHOULD BE RECALLED

- VMWARE SHOULD TAKE ACCOUNTABILITY AND WARN CLIENTS THAT THIS IS NOT A RELIABLE PRODUCT (YET)

So, I completely agree with you. After a couple of SRs, I had to give up. I don't get paid to troubleshoot "experimental" software. We paid good money for the VMWare product...it is a shame they ARE NOT LISTENING to their customers.

Kevin

[chriswaltham]

Has anyone heard anything from VMware regarding the possibility of fixes for these issues (or a 1.0.3 release)? I find that I have to either reboot my VDR VM every day or, at least, restart /etc/init.d/datarecovery in order for the VDR vClient plug-in to actually be able to connect to the VM.

[Stevester]

I heard late December something was supposed to be released from vmware concerning VDR

[XavierE]

While fixing something in vSphere I was told by a VMware support Eng that by late November VMware would release vSphere U1 and probably also an update to VDR

[kdc]

Hey

Just heard from vmware support that VMware vpshere 4 Update 1 will be out on the 19/11/2009 6:00pm PST he didnt know if it will have a update to vmware VDR

[KBuchanan]

Thanks for the heads up regarding the release.

I'm surprised the tech couldn't read/review the release notes to let you know if VDR was being updated.

Kevin

[einstein-a-go-go]

from Support!

Around the 20th of November VDR 1.1 will be released.

This release will fix a lot of problems and increase

performance drastically on certain operations.

I will send you another mail with the direct link once it

is released.

I managed to get going again, by stopping datarecovery service, removing *.lck files in the store, and restarting datarecovery. Now I need to workout why one VM backup stops at 73%

[Kiemo]

I wait with baited breath and hope this is the release we have all been waiting for!

Crosses everything.

[MKguy]

Well, seems like this and ESX4 U1 and View4 won't get released today (PST), but things have been further postponed:

http://virtualcloud.wordpress.com/2009/11/19/vmware-view-4-0-relase-date-postponed/

[kdc]

VDR 1.1 released!

Get downloading!

[KBuchanan]

I suggest, you STOP using VDR as a primary backup solution for production systems. There are enough problems in the forums to indicate the product has serious flaws and is highly unreliable.

STOP USING IT unless you are "testing VDR" as non-primary backup solution. If you are "testing VDR", then let me say "Thanks" - because I can't spend the time it takes to beta test it and deal with the headaches of it not working.

Kevin

[einstein-a-go-go]

We have just implemented vSphere 4.0, and very happy that VMware had incoporated a backup product VMware Data Recovery. The implementation was installed 8 weeks ago, VDR has been running perfect with no issues, until the weekend. The backups just hang, integrity and recatalog no longer work, it states the store is locked and we need to remove damaged restores, but no restores are marked as damaged, we have error -2240.

We raised an SR today, and the VMware Support Repesentitive told me VDR was SHIT! (his words), and we should use another third party product e.g. Veeam or Vizioncore vRanger. He told me Support Engineers view VDR as experimental, but management view it as production.

I'll be trying to speak to my Sales/Account Manager tomorrow to try and get a handle on this, because we have no money to purchase Vizioncore/Veeam product, and don't won't to use third party addons, when we've paid-up for Enterprise, and we cannot use the backup application.

Comments, anyone know how I can cure the -2240 error?

[KBuchanan]

Sorry dude...the. Engineer was right. VDR is crap.

Kevin Buchanan

Chief Information Officer

Lexington Memorial Hospital

Lexington, NC 27293

Ofc: 336-238-4286

Fax: [phone]

[einstein-a-go-go]

Well it looks like we'll be speaking to VMware tomorrow, and finding out what's the situation, and trying to get some money out of them, for a third patry product to backup up our environment!!!

[vAdmin]

Go Go Go Einstein !

Let us know the outcome, as a vSphere Essentials Plus user, there is USD $ 2000 price difference with the lowest vSphere Essentials license which is for HA and VDR features.

Kind Regards,

AWT

[KBuchanan]

Good luck...they "owe" all of us. At the very least, they should recall the product and provide some tangible service (i.e., extra months of support, extra license, etc...)

Kevin

[einstein-a-go-go]

fed-up with lack of response......

after posting a tweet today about VMware VDR, all **** has hit the fan, and the case has now been escalated!

http://twitter.com/einsteinagogo

I'll let you know how we get on...

Regards

e

[abbasi]

Are you having the same issues with VDR 1.1 as you did with 1.0x?

I haven't decided if it is worth my time trying to use 1.1 after the issues I had with 1.0x

[alexlie]

Had a lot of issues with 1.0.x - 1.0.2 wasn't running at all.

In contrast 1.1 works like charm. Having it running since the official release I have backed up about 30 machines, did several restores (complete, clone, single file) without any problems.

New GUI File Level Restore (FLR) for windows is nice (but slow).

I really hope things stay like this and I finally got a nice backup solution for my system.

Just in case of questions: I did not upgrade the VDR appliance nor reused the destination store. Since the previous version wasn't running I simple did a clean install from scratch.

[KBuchanan]

Time will tell.

In my experience with 1.0x versions, it seems that after 1 week (give or take) is when the problems would start happening.

Kevin

[Mirko_Huth]

Thanks for the info. Please keep us posted. I'm very interested if it still works for you after the first full integrity check starts...

That was always the point where my issues started with the older versions.

Regards

Mirko

[alexlie]

I don't remember when the "full" integrity check should be running, but the daily checks run smoothly for both destinations (900 GB + 1 TB).

[Stevester]

Happy Thanksgiving Everyone,

Has anyone performed any full VM restoring yet? How is restoring capability?

Stevester

[Kiemo]

Works fine for me. Have run restore rehearsals on several VMs including my test AD and SQL boxes.

Only thing I'm waiting to see working correctly is the Reclaim process.

[Stevester]

Okay!!!! However would you say that VDR 1.1 is ready for primetime? Or enough time hasn't passed yet?

Stevester

[Kiemo]

I can't honestly say that. As I've said in some other posts, my "gut feel" says this is the release we have been waiting for but I won't sign off on it untill I've had every aspect working reliably for a month at least.

If your 1.02 install is stable and mission critical. I'd veer away from upgrading just yet. If you've got the luxury of still being in a testing phase like me then go for the upgrade now.

I started a fresh with a new dedupe store rather than trying to bring over my old 1.02 store and I think this is a good idea as I've heard about issues with people upgrading their 1.02 data store.

Hope this helps.

[einstein-a-go-go]

With the help of VMware Support (Many Thanks to VMware), we found a <vm-name>.ctk.vmdk which was corrupted which was causing the backup of the VM to hang at 72/73%, Once this files had been removed, I'm now running VDR 1.1 as sweet as a nut, with no issues.

Backup and Restore are running fine. I have a 100GB local vmdk used as the store, mounted on a local vmfs (mirrored pair of discs) in my server. The VMs in my environment are shared from an iSCSI MSA2324i (SAS disks), 1 LUN per VM. At present we are using software iSCSI to do these operation over dedicated gigabit storage VLAN.

I much prefer the totally integrated solution of VDR, in vCenter for our clients with little VMware knowledge, rather than use third party products.

One happy VMware vSphere 4.0 Enterprise customer.

e

[KBuchanan]

Sounds promising...2-3 months steady and reliable usage will be the litmus test.

Kevin Buchanan

Chief Information Officer

Lexington Memorial Hospital

Lexington, NC 27293

Ofc: 336-238-4286

Fax: [phone]

[Kiemo]

Just to update my concerns above. The reclaim process is working and removing backups each day that match my removal criteria.

If its still working this well in early Jan 2010 I'll be a happy man!

[einstein-a-go-go]

From my tests VDR 1.1 does seem faster and more responsive, the VDR appliance has changed from Centos to Red Hat Linux (64bit I think!).

But I still have the hanging issue with one VM, it sticks at 73%, and then stops all the other jobs from running.

I managed to fix all the damaged restore points, and integrity check not running, by removing all the *.LCK files from with the datastore, and *.cat indexes. After this I re-ran datarecovery service, and was able to remove/delete (mark for delete) damaged restore points.

But with VDR 1.0, a few VMs still hung. I upgraded to VDR 1.1, and the same issue occured. I've since, moved all the VMs into seperate backup jobs, Brought all the different (16) backup jobs to compliance, and then moved all VMs back into one job. (Upgrade to VDR 1.1 was under VMware SR advice).

But I still have one VM, that faults, and sticks at 73%.

I don't know, why after 7-8 weeks it just stopped, and they all hung, and I had major issues, unless it's the store, thats getting bigger, but we're only small installation at present, 16 VMs, and 30-40GB, out of 100GB de-deuplication store.

The real issue here, we do not have any more cash to spend, after spending on VSphere 4.0 Enterprise + SAN, and we had hoped that VDR would do the job as backup, plus it's integrated as a total solution, rather than addon, Third Party products. We also don't have any cash left to purchase Vizioncore/Veeam.

We much rather the VMware on VMware approach, rather than Veeam or Vizioncore on VMware, if you understand, it keeps it simple for the us.

[KBuchanan]

So they escalated it. Now you'll get to hear from a "Senior Engineer" that "it still sucks". If they really can fix your problem...then...why don't they get somebody to fix the friggin' thing for EVERYONE???

Ok - so that is rhetorical. If they could fix it, they would. So the obvious conclusion is that THEY CAN'T FIX THE PROBLEMS WITH VDR.

Kevin

[einstein-a-go-go]

At least VMware Support are talking to me know, and I'm thankful for that. I'm concerened at the VMware Engineer comments though, and I'll be asking questions about this.

We will see what they find.

e

[vAdmin]

LOL.. ... same feelings here too Einstein :-o)

I wish that I can fully trust VDR than having to get 3rd party apps. to do the backup, this is such a shame to VMWare and also unfair since they close / limit the replication API in ESXi

Kind Regards,

AWT

[vAdmin]

To All,

Yes the VDR v1.1 is here :-o)

http://downloads.vmware.com/download/download.do?downloadGroup=DATARECOVERY11

it is Latest Released

Version:

1.1 |

11/19/09 |

207380

Kind Regards,

AWT

[admin]

For more information about VDR 1.1, take a look at my post

http://communities.vmware.com/thread/243436?tstart=0

[chrisaug]

Good information. Thanks.

[at12345]

We have this exact problem as well. It works well for a few weeks and then falls over. It then takes days to recatalog/integrity check but never finishes.

We cannot let the integrity check run indefinitely so we have to start again with a clean repository.

[chriswaltham]

Has anyone tried swapping between CIFS shares and directly-attached (i.e., VMDK) shares as destinations? Today I am getting yet another error message; I'm trying to run an integrity check on my destination and it's failing with error -2241 (Destination index invalid/damaged.)

[adisarro]

I've been using 1.1 since it came out and i'm very impressed by this update.

I would now trust this in a production environment.

[chrisaug]

I've been testing it since it's 1.0.x release and while I can say the new version is immensely better, I'm still having strange issues. I find I need to restart the device every few days, and today I was backing up 4 test servers and I noticed that the VDR VM just rebooted itself in the middle of a backup. The report said "Task terminated unexpectedly, possibly due to a power failure or system crash".

[Stevester]

Well the 1.1 verison is what is out now and from all reports it is stable, however i still feel though that VDR in general still lacks the nice robust features that its counterparts have such as email notification, an ability to perform FULL/Incremental backups on a cycle basis. The VDR 1.1 is a step in right direction as must "crawl before you can walk". Until that happens, I think I will stick with GhettoVCB.

Thanks

Steve

[adisarro]

I am using VDR just for daily backups. Just as something extra to have and certainly easier for file level restore.

For the main backups i'm using vcb weekly/monthly to tape.