vDP - Sick of being a beta tester

[SCX]

I am really tired of VMWare releasing half-finished software and expecting end users to be beta testers. First it was SSO (what a disaster that was/is), and now vDP - what is going on with this appliance? It's a complete resource hog that crashes constantly. Why does it need 4 CPUs? What on earth can it be doing that needs so much? Why does it take 10 minutes to boot the appliance? Why is installation such a mess? How come when I try to start backup jobs, half the time, nothing actually happens (apart from a snapshot being taken)? How come you are forcing me to use the webclient to access it?

The whole 5.1 release reeks of being rushed and I'm thoroughly unimpressed.

[GSparks]

I am sorry to hear that you are frustrated with your experience with VDP so far.

We would like to work with you through your issues so that you can see the value that VDP can add as your backup solution.

In regards to your questions, here is some information to provide some insight into how the appliance works.

Why does VDP require four CPUs?   

Several of the activities performed by the software are CPU intensive.  Examples are garbage collection, the HFS check and backup and restore hash calculations.  Without adequate CPU resources, these activities would cause performance on the system to degrade below what we deem acceptable.

Why does it take 10 minutes to boot the appliance? 

During the initial VDP appliance setup, we are performing many of the following steps –starting up core, file system and management services; network reconfiguration; keystore and SSL setup; registering with the vCenter and proxy clients; synchronizing of passwords across all subsystems; and performing an integrity check, which creates a checkpoint and validates the checkpoint.

Why is installation such a mess?

Can you please provide more information as to what was problematic with the installation?  Your feedback is valuable to us.

How come when I try to start backup jobs, half the time, nothing actually happens?

This sounds like a configuration problem.  Would you be interested in opening an SR with support?  I’m certain we can solve this issue for you.

How come you are forcing me to use the web client to access it?

There are a variety of reasons why we are moving to web based management UI.  In our user testing, we had customers who initially were hesitant to move to the web-based client, but after using it for a bit, they felt it really makes sense.  Plus, in multi OS/browser environments, the Flex UI is a real treat.  Any particular feedback you have is welcome here as well.

Thank you,

[Maurice Volaski]

I would like to give you some feedback on how 5.1 and VDP is a major downgrade in terms of user experience.

First off, the program and the data are one piece. I get fixed sizes, which was not the case in VDR. Worse, do you not expect anyone to implement this on simple mirrored disk arrays? That is, the 1 TB ova requires a 2 TB disk and the 2 TB one requires a 4 TB one because it requires slightly more than the size of a 3 TB disk's capacity. Of course, if I wanted anything larger, I have to have multiple instances of VDP. What was simple has now become inordinately complex.

Second, for my simple installation, I have a private network and I just used a hosts file and it worked. Now VDP is hardcoded to ignore it (to ignore nsswitch.conf in the VDP appliance) and to use DNS exclusively, so I had to set that up from the ground up. Doesn't that violate POSIX or something to ignore nsswitch.conf?

Third, when the installation prompted me for my vCenter username and password, it rejects me. I recall that before (after doing a simple install all of vSphere 5.1), I had to add explicitly add the Administrator (the vCenter Administrator user) as a user to SSO before the web client would see my vCenter. But then I had to at least prepend the domain, that is, system-domain\, to the username before it would be accepted by the VDP installer. I believe I had to add a separate user other than the Administrator for that, but I don't even recall. This should have all be done by the installer automatically.

Fourth, what's up with the crazy password policy for VDP, which is on top of a different crazy password policy for the SSO?

I just played a little with the web interface...

Fifth, I find it more confusing than in VDR. In VDR, there were colors and icons and I could see clearly in events what had taken place. Now the listing in reports is totally generic with all the text the same size and colors. For example, the state of the machine is right there alongside with the backup status. Who cares about the former? I really just want to know which machines failed. They should be in their own list at top along with a detailed reason field to explain why. (I have found that I can manipulate the columns, but it's limited and why not just set them up optimally to begin with?)

Sixth, I may be misunderstanding some things here, but I don't see a way in the backup configuration to select individual VMDKs. When I click a triangle for a particular VM, it shows no further details.

Seventh, in VDR, if I ever had to restore the appliance itself, I could so easily and just point it back to the storage. It's not clear how to do with VDP, which installs only with its storage.

Lastly, I can't speak much for what's under the hood in VDP. VDR would often fail with cryptic errors and one desirable feature was some level of file restore. I think it would have been better to concentrate the effort on just those things and keeping the interface inisde vCenter mostly intact.

[SCX]

I agree about what (to me) seems like overly obstructive and confusing security configurations in SSO. Then again, we are a SMB and need nothing more complex than a single "administrator" user that should be given permissions to run anything. Having to explicitly set permissions for SRM, vCenter vDP etc by using the SSO admin account is cumbersome at best.

And yes, what is with the ridiculous password policy in vDP? Exactly 9 characters, no special characters, one uppercase character etc? Fail.

There is no way to backup individual VMDKs - it's by virtual machine.

I don't care how much you try to convince me how much work the machine is doing under the hood, 10+ minutes for a reboot is unacceptable.

Regarding the web interface, it is just "ok". What about showing me the space taken up per VM? I can't seem to find that anywhere.

The "Filter" functionality on the "Reports" tab is unfriendly and buggy.

Anyway, the best thing I can say about it is that it is "free". Hopefully, future updates will make it a bit more user-friendly.

[RParker]

I got news for you Greg.. VDP / VDR has been in BETA since it has been out, we have BEEN complaining about this lousy software since DAY 1!.. NOW you are sorry to hear we have a problem?!?!?!?!?!?!?

Uh... you don't pay attention much do you?

VDR has been buggy since inception, apparently VDP isn't any better NOW either. I will not recommend it to ANY vWare customer.

FIX IT!  Tell VMware engineers to ACTUALLY USE it, instead of programming .. and as the OP put it.. stop MAKING USERS BETA TESTERS!

I don't see how it's SO blatanly obvious it's such a worthless, horrid software that VMware refuses to ACTUALLY see for themselves..  It NEEDS to be FIXED NOW!!!

[Josh26]

Greg Sparks wrote:

Plus, in multi OS/browser environments, the Flex UI is a real treat.  Any particular feedback you have is welcome here as well.

Thank you,

"Flash" is, by definition, the opposite of "a real treat".

Sensible security policies ban it on servers. Being "multi OS" would have actually been reached if the interface was HTML5, but it isn't. Instead, it's a proprietry system, supported on the platforms Adobe chooses to support.

It's clunky and crashes on a regular basis. There's no value in filing an SR because it's usually a safe assumption that the response will be "that's an Adobe crash and we aren't responsible for it".

[Litejk]

I am also evaluation vDP and I have used vDR before.

1. Installation is painful, and not very bullet-proof. In my case, I could not register the appliance with the hostname of the vCenter sever (even if all the records where ok and verified), I had to use the IP address in stead. Same goes for SSO servers hostname, it would only accept IP address.

Second one, in my test, I had to configure vDP 2 times (with 2 reboots), before finally I could log in to the post-setup status and settings pages.

2.  Way to high CPU usage. On the testserver it runs on now, it constantly uses between 4,500 MHz and 6000 MHz on a testbox with single CPU with quad core running at 2,5 GHz, which means that half of the CPU's in average are running at max peak constantly. Very strange patterns in performance tab also. Note that the vDP appliance only have done 1 single backup of a small Linux VM at around 6 GB (using a 1 TB datastore) . I cannot think of any reason for why the appliance needs to do a heck of maintance and garbage recycling constantly for so small amounts of data.

[url=http://bildr.no/view/1302674][img]http://bildr.no/thumb/1302674.jpeg[/img][/url]

[url=http://bildr.no/view/1302675][img]http://bildr.no/thumb/1302675.jpeg[/img][/url]

[url=http://bildr.no/view/1302676][img]http://bildr.no/thumb/1302676.jpeg[/img][/url]

[url=http://bildr.no/view/1302677][img]http://bildr.no/thumb/1302677.jpeg[/img][/url]

I see that there is a lot of java processes going crazy sometimes, with peaks up to 94% of total CPU's on the appliance.

3. The Web Client is way to slow, but that goes for all administration of vSphere when using the Web Client. The old fat client is much more snappier.

Why in the earth rely on flash for creating an admin interface? In my mind, even standard simple HTML (not HTML5) would be enough to create the UI.

o

4. File level restore requires you to do this ONLY from a server that has only been backed up with vDP. That means, if I need to restore files from 2 servers (ie REDHAT1 or Win2K8R2), I have to login to either those 2 servers or other servers that has alreade been backed up. I cannot do this from a client computer (for example my own laptop). That means to use the VM Console, RDP or VNC and then start it. Why in the earth is that a show stopper? The old vDR did not have that requirement and to me it seems that someone who designed this solution at VMWare seriously need to basic system architecture again. Since vDP requires the client to initiate an advanced login to be a VM it self (pushing application logic to the client in stead of keeping it middleware/server layer), I think it makes it very cumbersome to do FLR. After all, who is running GUI on a Linux server anyway nowadays? And on Win servers, I never install any Flash Player and I'm not happy to install the Client Integration extensions either on production servers.
In my mind, the vDP appliance should not care from which client you are coming from, as long as the client have access to vCenter also.

Edit: 5. The Web Client is buggy when it comes to international keyboards. For example the sign '@', which in Norwegian (and other Scandinavian keyboards) to press Alt GR + 2, don't work, it just forces the client to go back on step. Only solution to overcome the limit is to write any text in notepad and copy it (for example when filling out email addresses on Reports tab)

[RParker]

Let us also not forget the 100 VM limitation per instance of VDP, it will do 8 concurrent backups (not settable) per instance, and it's limited to 1TB EACH datastore attached to VDP instance, and you can only have 2!

VDP is just a completely useless, and utterly buggy.  I am glad you posted what you did... don't let up, keep on them..

[cojo71]

1.  I really don't understand why it required almost 1TB for 500GB backup drive.  Putty into VDP, I see you allocated 77GB for /space(include avamar and /home) and it is using 1.4GB.  Most of the remaining space are allocated to /data01, /data02, /data03 and I see backup are stored in these folders.

2.  Flash player is one of the buggies software with endless updates on earth and I have to install it on the server before I can use FLR.

3.  It took 18m21s to restored a single 124MB file using FLR?  CPU utilization hit 100% during the restore process.  I can't imaging what it will do to my server if I have to restore a bunch of files or a very large file.

[VMware Community Member]

I also upgraded from vdr to vdp and I'm really disappointed with the upgrade. I spent 2 days configuring vdp and experienced the same problems mentioned in this discussion (sso problems, dns problems, configuration problems, ...).

I would be nice if Vmware added a VDP administrative interface in the vSphere Windows client with FLR support. This would solve a lot of annoyances for adminstrators (Flash requirement on a server for FLR, clumsy web interface, ...). When I go to settings - plugins in the vSphere Windows client the VDP plugin is already listed so maybe this is already (partially) implemented ?

Vmware, if you are listening to your (paying) customers you should at least give us a timeframe in which these problems will be resolved.

[krismcewan]

As a consultant having recomended our customers to use VDP we are now in the embarrasing situation of having to tell them its not suitable for production use.

We have NEVER managed to get the Webclient to connect to the appliance. (outstanding SR with VMware on this 6 weeks and counting)

Based on everyone elses experiances We have NO confidence when we do get it working that it will work.

Having a VM with internal storage to back up to is UNNACEPTABLE.  95% of customers have a single SAN that they use for VM's they do not want to backup their VM's to the same SAN.  Not being able to use a CIFS share that is either external or on cheap storage that can be dumped to tape and located offsite is not a backup solution.

The boot time is UNNACEPTABLE.  Being on the phone with VMware Support for 4 Hours half of which is waiting for a VM to Boot up is a waste of my time and the poor Support person at Vmware.

Worst of all Vmware Support do not have the skills to resolve issues as they seem to be Escalated to EMC Avamar team.

This is a serious hit to the Reputation of VMware software.  Stop letting the Sales and Marketing team drive innovation.

[PlanglowLtd]

I couldn't agree more on most of what has already been said in this thread.

Currently I'm re-installing VMware Data Recovery to backup to an external NAS for disaster recovery. Other than VDR the only other options offered to me so far have been £1000+ per host licensed products... forget that!

[edinburgh1874]

Are there any fixes for the Java processes using large amounts of CPU?

I've just deployed a VDP appliance, with no backup jobs and Java is instantly using 3 GHz....the VMWare KB just states that a reboot is the fix, the issue just reappears.

Cheers

[Buschfunker]

Same Problem here.

8Ghz of CPU usage.

Why is there no statement from VMware itself?

This backup solution works very unreliable.

[edinburgh1874]

Hi All,

I actually received a call from VMWare which helped us understand the issues with VDP a bit better.

VDP will use as much resources as it possibily can to do backup, maintainance and integrity jobs - we seen it use 10GHz and 32Gb during a job.

To counter this if it's in a DRS cluster, you can limit the CPU/memory usage, the jobs will complete but will take a much longer time.

VMWare don't set any limits out of the box, in order to make the jobs complete as quickly as possible.

The first job will consume a massive amount of resources, but subsequent jobs will be less intensive.

Backup jobs can appear to be running all day, but are not actually touching the source - it's running dedup on the backup data.

VDP is based on EMC Avamar, which is usually deployed on an appliance dedicated to backups so is used to having it's own resources.

So, given this, we deployed it on it on an old HP G1 with it's own storage and it's much happier.

Also, the appliance must be allowed to complete the blackout window - so this may mean rebooting, etc, during a backup window...perhaps some folk are doing maintaince in the blackout window causing integrity issues.

When I first tried out the product, I thought that all maintainance was in the "maintainance" window and that the blackout window was clear...I didn't read the manual well enough.

[CraigD]

We setup (EDIT) VDP and have it backing up two servers in a test ESXi environment.  We added the vCenter appliance and the VDP appliance to our existing test ESXi box in order to do this.  I did not want to set up VDP on our production vSphere 5.1 system, and I do not have the free space on our production SAN.

I hate it that I cannot "point to" external storage from the appliance.  Instead I must carve out enough space for the pre-allocated appliance.  Also, if I choose wrong (i.e. a smaller appliance than what I really need), there is no way for me to move to a larger appliance and keep my backups.

The FLR process is very cumbersome and not very practical.  I have tested it and it does work, but wow, what a chore to get a file off a backup!  This must be improved.

Lastly, there really needs to be a way to ship the backups offsite.  I would love to use Amazon or some other cloud provider.  Maybe someone can outline for me how to do this... or maybe there isn't a way.  From what I have read, VDP (and its data) just isn't "portable."

What I really want (30,000 foot view) is a system that will run full VM backups that are both runnable and accessible at the file level (for file-level restores).  I want to be able to store these backups on local storage and also transfer them to offsite (preferably to the cloud).  Lastly, I want to ensure that in the event of a disaster I can connect these backed-up VMs to a completely new VMware environment (if necessary) and crank them up and have them run happily.

Anybody got tips?  Veeam, perhaps?  Can it be done with VDP?  Is it on the VDP road map?

VDP has so much promise, conceptually... it just needs to actually work.

[edinburgh1874]

Offsite backups is potentially possible using vSphere replication of the VDP VM...I'm looking to implement this but am still building the DR site.

The other option is hosting VDP on an NFS datastore, then replicating that.

[jb42]

I agree in principle, I wish it was easier. But after struggling with it for a while, I think it can be done.

I'm hosting VDP on iSCSI zfs volumes. ZFS has snapshots and replication. Replicated appliance is coming on line as I type so I'll report back with results. Here's another DR method utilizing ZFS that doesn't include VDP: http://blog.laspina.ca/ubiquitous/provisioning_disaster_recovery_with_zfs. And here's a blog post showing an example VDP replication technique, though not the one I've used: http://blogs.vmware.com/vsphere/2012/12/recover-replicated-vdp-appliance.html.

If you look into ZFS, OpenIndiana and Napp-it are good searches to start with.

[gravesg]

I've have not one, but now TWO VDP appliance unrecoverable crashes.

Seriously, how bad can this product be?

[edinburgh1874]

I'm thinking of ditching it, we had a Linux VM that hung during a VDP backup and then reported "Unknown Internal Error" on trying to reboot it.

We weren't able to restore any of the backups.

Such a shame, since VDR was quite reliable.

[edinburgh1874]

I've definately scrapped it...

I found that although the backup job we are running shows that it is running every night, successfully, the latest restore data is one month old.

No errors in the logs to reflect this, VDP integrity checks are running fine.

There are 7 VMs that always fail backups, again not sure why. On rebooting these VMs, they complain about the .vmx file being in use...I can only reboot the VMs by SSHing into the host and deleting the .vmx~ file that is left over.

I assume that VDP is locking the VMs somehow.

[edinburgh1874]

For anyone interested in alternatives, but have a low budget here are two great options..

1. Quantum vmPro - this has the same functionality as VDP, if your enviroment protected size is under 1Tb it's free as well. Very easy install as an appliance, low resource usage and seems to be very stable. It's missing dedup, but Quantum also offer a storage appliance called DXi v1000 which dedups data stored on it's NFS and CIFS shares. Again, up to 1Tb of raw storage is completely free. Great dedup rates and some fantastic reporting tools built in.

2.  Trilead VMExplorer - our enviroment is larger than 1Tb so we ended up going for this as a VDP replacement, we can't afford Veeam or any of the other large names. Licencing is very fair - £450/$700 for the whole enviroment, 1 year of updates. Only disadvantage is that it doesn't offer dedup, but if you team it up with the free DXi v1000 appliance I mentioned above you can obtain this functionality if your post dedup size is under 1Tb.

If anyone has used the above tools and has any comments I'd be happy to hear them,

[jjharrison81]

Agree with most of the above.

This solution is not even close to being ready for enterprise.

It's an embarrassment to VMWare and an embarrassment to those of us who recommended VDP to clients.

[g754ww]

this solution is pure garbage..Vdp isn't ready for primetime.I too had to reinstall many,many times.. EMC owns VmWare,so that's why Vdp doesn't work - they want you to buy EMC legato networker to backup your VM's..pure and simple..

you can try VirtuallyGhetto.com they have a somewhat crude but effective backup , vghetto, that works well..

part of the problem with Vmware is that to get VM's out of esxi it is very slow, it's like they've tuned down the scp sftp and cp programs , i.e., purposely made them work more slowly than,say,cloning..thats just my experience..it shouldn't take forever to copy a 45GB VM at the command line

cp <vm> /datastore_name, especially if it is already locally NFS mounted or iscsi..

someone else mentioned the ZFS replication idea - that's a pretty good approach..you could try Freenas..I use it with an 64-bit machine, with two disks and a 2TB zfs volume..works well..

on a windows box with enough HDD space, you could  install Windows services for unix, nfs server,and export that windows NFS drive as  esxi datastore, and backup to that , which is something I do because this windows box is ,in turn, a member of our nightly tape backups..

on windows 2008R2->,you can make a local hard disk an iscsi target, and mount that as an esxi datastore as well..

I've also installed Bacula , the open source backup software project ( onto a Fedora VM) it works great! easy to set up, backs up to disk or to tape, windows,linux,unix,etc..

comes with a web-based gui..restore work well..was seeing double the I/O using bacula over iscsi vs. bacula over NFS..

[MartinSvec]

Well, can't imagine how somebody can pay money for VDPA. We gave VDP another shot this week and there are still no serious improvements. Below is a list of issues and annoyances we've found during our tests:

• VDP starts 10 minutes (sometimes 10 hours if you've a bad luck). It takes hours to solve issues that require reboots.
• Terrible CPU/power consumption 24 hours/day.
• Backup of powered-off VMs always fails in our production cluster. (But works fine in out lab.)
• FLR does not work in our production cluster. (But works fine in our lab.)
• FLR does not support ext4, XFS.
• 5000 files limit in one FLR restore operation.
• FLR client requires Adobe Flash, no Linux-friendly command-line FLR tools.
• FLR client must run inside VM which is backed-up by VDP. If you want to use one FLR client for multiple VDPs, its VM must be backed-up by ALL those VDPs. (Oh my God...)
• Upgrade from 5.1.10 to 5.1.11 fails.
• Cannot take the dedupe storage and mount it into a new appliance if the original one is damaged.
• Cannot backup individual VMDKs of a VM (Should be fixed in next major release.)
• Cannot specify backup jobs start times, all jobs start at the beginning of the backup window.
• If VM backup fails, VDP doesn't retry it after 30 minutes like VDR.
• No useful information what happens during the backup process (VMs which are just being backed-up, % of progress, etc.)
• Terrible logging, no useful logs available in Web Client. Looking for a reason why VM backup failed is a nightmare.
• Reports in GUI are simply useless compared to reports in VDR.
• All report times are in US AM/PM format only (probably a feature of the Web Client).
• ...etc, etc.

VDR had it's own problems but compared to VDP, it was an acceptable SMB backup software.

I must ask myself how many people work on VDP. One or two engineers in their free time? Definitely no more... Wouldn't be better to fix dedupe/integrity issues of VDR instead of replacing it with such a garbage? Together with SSO, this is another disaster that comes from the hasty integration of EMC products into vSphere.

And no, we really don't want to waste our time by opening SRs regarding some of the issues if there are so many design-related problems.

[Buschfunker]

So Mr and Ms. VMWare......DO YOU MIND YOUR CUSTOMERS PROBLEMS??????? WE NEED HELP!!!