Ways to convert VMDK files

[beebee201110141]

Hi,

We have recently had a server failure and I have decided to rebuild the server and put ESXi 4 on it.

However, the backups I have of the old machines are on NTFS formatted drives (not USB, they are in our backup machine) which means that they are unable to be mounted (as far as I can tell) under ESXi.

The problem I have is that the VMDK files I have are the older version so not compatible with the curernt ESXi. I know I can convert them using vmkfstools (with the -i switch from memory) but I have two problems with this. Firstly I have to get them on to the ESXi server in the first instance, and since we are looking at about 5Tb of data it is going to take FOREVER (been running for 2 days now - still nowhere near complete) and the second is even if I do get them onto the server, I will need another 5Tb of drive space to convert them into, which I don't have.

So my question(s) :smileyhappy:

1. Is the really not anyway to mount an NTFS drive (not USB)  under ESXi - that way I could just convert from the backup drive to the datastore (nice and easy)

2. Is there an offline converter for VMDK files?

3. Is there a way to change the NTFS drive to a readable format by ESXi (ext3/resierfs/something) without losing the data

4. Is there a way to optimise the network transfers maybe?

5. Is there another solution I could look at

Many thanks in advance

Bob

[idle-jam]

you will have an old VMDK, you could then use vmware converter to pick on it and convert it to a workable format on ESXi

[beebee201110141]

Hi,

Firstly thank you for taking the time to respond.

I have the stand alone converter but that appears (I could be using it wrong) to want to pick up virtual machines and convert them rather than virtual disks. If I am missing something, let me know and I will check it out! Or is there a stand alone VMDK converter I should be using?

When the server died, it munched all the VMs, but fortunately we were able to recover most of the virtual disks (different RAIDs). Hence my predicament!

Actually I have hundreds of VMDKs - all split into 2Gb files. My big issues are that even if I do wait long enough to get the files across to the server, I am going to find it extremely hard to find the necessary space to convert them on the server. I would much rather convert them from the backup drive TO  the space on the server and out of preference not over the network as it does appear to be terribly slow.

Do you know of any way that I can get ESXi to see the data on the NTFS drive and then I can just vmkfstools the conversion locally on the server?

Thanks

[idle-jam]

you could try this, using a vmware converter select source as vmware workstation and point to all the VMDKs destination (could be file share) and destination is the ESXi.

[beebee201110141]

I will give that a go this evening and let you know how I get on. Many thanks

[beebee201110141]

I tried using the converter and selecting the workstation, but it only gave me an option to select vmx, not vmdk.

I'm going to have a go at mounting it through NFS tonight instead.

Will let you know how I get on. If you have any other ideas on how to approach this - I'm all ears :smileyhappy:

[beebee201110141]

So close!

I can successfuly mount the NFS as a datastore under ESXi, but my problem now exists with cloning the exisitng vmdk's

Using vkfmstools -i %path%/firstdrive.vmdk %path2%/newdrivename.vmdk I now get a

Destination disk format : VMFS zeroedthick

Failed to open 'firstdrive.vmdk' : Reached the host's limit for open files (40)

Is there a way to raise this limit - I have 476 of these files for this drive!

More info : Server is running off a backtrack live CD with the drive mounted rw by the nfs-server.

[beebee201110141]

Tried changing the limit for nofiles to 1000 using ulimit -n 1000 and now I get a segmentation fault.

Grrrr!

Please help :smileyhappy:

[opbz]

are you sharing your ntfs using NFS?

not sure if this will help or not but

how about booting your server with the ntfs file system using something like ubuntu (it supports ntfs) then use scp to copy vmdks to local stores  on esx server and then finally do vmkfstools -i to import them...

to be honest  I can think of other ways of doing it but none are pretty and most are pretty convoluted..

[beebee201110141]

Hi,

Again, thank you for taking the time to respond.

Yes, I was sharing the drive over NFS. My reasoning for doing this is that I don't have enough space to be able to copy all of the data over, and then convert it, all on the same server. Actually, I could probably manage it but it would be a great deal of hassle to try and achieve this.

So my reasoning was if I could wipe down a datastore, use the NFS share over the network, it would allow me to convert from the backup drive to the datastore and eliminate the need to juggle the space on the ESXi server. The ransfer times over the network (for some reason I have yet to fully investigate at this stage) are horrendously slow. To move one of the drives over to be converted will take in the region of 3 days :smileysad:

I really thought I was nearly there last night! I can mount the NTFS using NFS (under a linux live CD) on the ESXi server, get to a console on the ESXi and then run vmkfstools -i and then let it run over the Easter bank holiday here to do what it needs to do. Then I hit the limit issue followed by the seg fault.

At this point I am more than happy to take any ideas on how to proceed - however Heath Robinson-esque they may be :smileyhappy: I'd even consider an alternative to ESXi to be able to restore this data.

idle-jam suggested using the standalone converter to convert the vmdks but I cannot find a way to open vmdks as opposed to vmx. There must be someway of converting vmdks not from the ESXi console. Are there any error logs I could be looking at as to why the vmkfstools seg faults? Is it worth grabbing a trial of something (vMotion?) that will allow me to do this? Is it worth installing a version of vmserver, creating a vm, adding the vmdks to that machine and then using the convertor?

I'm willing to consider any solution at this point. Even moving to Hyper-V or Xen (which has a live CD available - is that worth looking at to convert the drives and then send them to the datastore using the convertor?)

Once again, thank you for taking the time to help - it is really appreciated.

[opbz]

ok pretty messy stuff :-)

one issue I noticed from earlier was that you are talking 5 TB.... there is a 2TB limitation on VMFS. (pretty sure you know this :-) ) Also the vmkfstools -i command consolidates all the vmdks and creates 1 big one.... in other words more space needed.

One option you have is to install workstation and try mounting the disks then running standalone converter inside the VM. Thing is this will be slow....

other options

if you have space on your NTFS machine you could potentially create an fat32 partition. copy files then you have options....

1: boot to live linux and use mount on the ESX  server to mount the files on the fat32 fie folder...

2: create yourself a boot ESXi usb disk and boot to it... then run the vmkfstools from there...(might give you issues as it wants to run on vmfs volumes)

other options.....

setting iscsi targets to on the windows server and try mounting these on ESX server (again 2TB limitation) on windows this is messy on linux not too bad I have done it using centos. But there I created new disks. This is pretty messy requieres coping files to iscsi target then mount target to esx server ....

do you have backups of the VMs? If you then setup new vms and try restore to them...

too be honest most of these options are pretty messy and not guarnateed...

[beebee201110141]

>ok pretty messy stuff :smileyhappy:

It's never easy :smileyhappy:

>one issue I noticed from earlier was that you are talking 5 TB.... there is a 2TB limitation on VMFS. (pretty sure you know this :smileyhappy: ) Also the vmkfstools -i command consolidates all the vmdks and creates 1 big one.... in other words more space needed.

It is a total of 5TB of data, fortunately, else I would really be up a certain creek without a paddle, the old VDs are <2TB in size.


One option you have is to install workstation and try mounting the disks then running standalone converter inside the VM. Thing is this will be slow....

Willing to try anything at this point :smileyhappy:

>if you have space on your NTFS machine you could potentially create an fat32 partition. copy files then you have options....

Space issue again, but might have to look at begging/borrowing/stealing a couple of drives to do this with!

>1: boot to live linux and use mount on the ESX server to mount the files on the fat32 fie folder...

Definitely worth looking at

>2: create yourself a boot ESXi usb disk and boot to it... then run the vmkfstools from there...(might give you issues as it wants to run on vmfs volumes)

Interesting - I didn't know you could make an EXSi boot disk. Do you have a link handy for the download or is it something you have to maek yourself?

>other options.....

All ears :smileyhappy:

>setting iscsi targets to on the windows server and try mounting these on ESX server (again 2TB limitation) on windows this is messy on linux not too bad I have done it using centos. But there I created new disks. This is pretty messy requieres coping files to iscsi target then mount target to esx server ....

I'm afraid you have lost me a little bit here. Could I ask you to give me a more detailed explanation? I don't understand even the first bit "setting iscsi targets to on the windows server " I'm afraid. :S

>do you have backups of the VMs? If you then setup new vms and try restore to them...

I wish - the server failure muched the VM RAID, but not the VD RAID.

>too be honest most of these options are pretty messy and not guarnateed...

It is certainly more options than I had a few hours ago. Something tells me I won't be enjoying the Easter weather too much :smileyhappy:

Thank you once again for taking the time to help out.

[beebee201110141]

No worries - found the instructions for making the bootable USB (thank you Google!)

I was wondering if you thought I might be able to use the trial of SvSAN for VMware to help me achieve my goal?

Thanks

[opbz]

easiest way to create a bootable usb of esx is to download the embedded version from dell...

http://support.dell.com/support/downloads/download.aspx?c=us&cs=555&l=en&s=biz&releaseid=R301605&SystemID=PWE_R710&servicetag=&os=EX41&osl=en&deviceid=25793&devlib=0&typecnt=0&vercnt=5&catid=-1&impid=-1&formatcnt=0&libid=47&typeid=-1&dateid=-1&formatid=-1&source=-1&fileid=453775

then use a dell server (might work on other servers but I onlty tried it on this ones) do an install of ity it will do it onto the usb drive. Then take out drive and boot your other server.

by the way have you tried stuff like winscp? Might work if your vmdks are not too large... also could use it along with md5sum windows checker  to ensure all is copied ok.

SVsan sounds very interesting... never used it though. I tend to use linux stuff as its ussually free..:smileygrin: or stuff like freenas...

main thing about this option is that you can first present an iscsi lun to your windows machine copy the vmdks and then present it to your ESX server...It just sounds pretty painfull though...

Other options I though of where...

BIG usb drive....formatted as fat32...

Windows VM on ESX server then use CIFS to copy vmdks to it..then virtual infracstructure client and browse/copy to datastores on esx server (should be fast as its local at this point) finally consolidate... Still very messy...

Personally by far easiest is copy to usb and then mount usb on esx server... other options are like trying to force a round peg in a square hole...

[beebee201110141]

Happy easter everyone!

Tried loads of things, including waiting nearly 4 days to copy over one of the drives.

Still getting a segmentation fault on the vmkfstools -i command

output is this :

vmkfstools -i vd1-media.vmdk Media1.vmdk

Destination disk format: VMFS zeroedthick

Segmentation fault

What error logs can I look into to try and diagnose this ?

Many thanks

[DSTAVERT]

What OS is on the NFS server? Do you have command line access? Please list the names and sizes of each VMDK that you need to convert. How were the VMDKs created?

[golddiggie]

Just for giggles, you could try vConverter from Quest Software (http://www.quest.com/vconverter/)...

[beebee201110141]

DSTAVERT

What OS is on the NFS server? Do you have command line access? Please list the names and sizes of each VMDK that you need to convert. How were the VMDKs created?

Currently running using a BackTrack live CD (4R2). VMDKs are called vd1-media.vmdk -> vd1-media-f476.vmdk. Each one is 2147221504 (apart from the last one, vd1-media-f476.vmdk which is 124518400) They were created under VMware Server several years ago.

golddiggie

Just for giggles, you could try vConverter from Quest Software (http://www.quest.com/vconverter/)...

I will look into that now - many thanks!

[HalfMuffin]

Whoa Whoa Whoa, back the truck up.

You can use VMware Converter to do this and save yourself a TON of headaches.

First off, what version of VMware were you PREVIOUSLY using to host VM's? VMware Server? VMware GSX? VMware ESX 2-3.5? VMware Workstation?

Secondly, Do you JUST have the VMDK files? or do you have the accompanying VMX files as well? If you don't have the VMX's, you have headaches on multiple levels. Converter needs the VMX file to know how many CPUs the VM had, what hard driver controller it had, etc to do a proper conversion. If you DONT have the VMX files (and just the VMDK's) AND you were running ESX 3.x, you can remake the VMX file by making a NEW vm on the ESXi 4.x server and attaching the old VMDK file to it. You will need to make these VMs as VM hardware version 4 (NOT 7) and take a "best guess" as to what type of hardware they had for network card, scsi card, etc. This might get you in the right direction very fast and would be easy to test on one of the smaller VM's if you had any tiny ones.

Third, What were you using to get backups of the VMs previously?

This might be easier to do offline, feel free to email me at jonmkohler AT gmail DOT com and I would be happy to give you a hand with this. Otherwise, post responses here.

Jonathan Kohler

VCAP4-DCA / DCD

[beebee201110141]

Hi Jonathon,

Over the extended break I have tried pretty much everything and ultimately I have gone the way that you recommended.

I installed VMWare Workstation trial and set up a new VM and attached the VMDKs to it then used the VM Converter Standalone to convert/upload the VMs to the ESXi server which has worked 99% (network card doesn't work correctly but I am going to change the OS on the VM now it is up and running anyway so a clean install should fix that problem).

Interstingly, doing it this way I managed to get all the data converted over and up to the ESXi server in less time than it took to upload just one of the drives (2 days compared to nearly 4 days for just one of the drives)

Unfortunately I did only have the VMDKs (VMX and VMDKs were on different RAID arrays and the VMX array is the one that died horribly) which was causing me, as you correctly predicted, headaches on multiple levels!

At least now I can access the data on the ESXi server, though not on the network (yet!) which means I am on the right path to getting this back up and running. Thank you for your reply, and especially for offering to help out offline which was greatly appreciated.

Thanks again