Converter 5 does not recognize LVM partition on Oracle Linux 6

[ibodog]

I've got two physical servers running Oracle Linux Server release 6.3 64-bit.  They each have a /boot partition that is ext4 and the rest on an LVM partition:

/dev/mapper/vg_cleveland-lv_root /                       ext4    defaults        1 1
UUID=3d37688e-305a-4974-9bb7-f02232b78790 /boot                   ext4    defaults        1 2
/dev/mapper/vg_cleveland-lv_swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0

vCenter Converter is only recognizing the /boot partition in the "Data to copy" section "Select volumes to copy".

If I go forward the conversion fails when trying to reconfigure the VM.  I've looked through the user guide and googled a bit but I can't seem to find why the ext4 LVM partition is not being recognized by Converter 5.

[Ivan Ivanov]

Can you provide task log bundle? You should switch the GUI to *TASK VIEW* and then right-click on the failed task and select "Export logs" command.

[ibodog]

I've created the bundle, but it looks like there may be sensitive info in there.  What should I be looking for?

[Ivan Ivanov]

In converter-worker log file there should be a XML and a log file generated by vmware-sysinfo binary which was executed on the source. There should be information about the devices and volumes detected on the source and possibly a reason for filtering anything out.

[ibodog]

Not much to go on.  There seems to be just one warning after vmware-sysinfo is run and all the other warnings I see later are probably related to not being able to find anything besides the /boot partition ([02748 warning 'Default'] DefaultValues::FindGuestOsDescriptor: Unrecognized vmomiGuestOsId=redhatGuest, falling to otherGuest).  Here's the relevant part (scrubbed of some data):

[06992 info 'Default'] [Converter Agent SysinfoQuery] successfully exec'ed vmware-sysinfo; result: <?xml version="1.0" encoding="UTF-8"?>

--> <sysinfo>
-->  <storage>
-->   <volume mountPoint="/boot" blockSizeInBytes="1024" capacityInBlocks="495844" blocksFree="402778" blocksUsed="93066" devicePath="/dev/sda1" fileSystemType="ext4" isBootVolume="true" label=""/>
-->   <disk type="msdos" controllerType="scsi" bus="3" deviceId="0" capacityInSectors="156301488">
-->    <device path="/dev/sda" major="8" minor="0"/>
-->    <hwGeometry cylinders="9729" heads="255" sectors="63"/>
-->    <biosGeometry cylinders="9729" heads="255" sectors="63"/>
-->    <partition number="1" type="primary" isActive="true" fileSystemType="ext3">
-->     <device path="/dev/sda1" major="8" minor="1"/>
-->     <lba start="2048" length="1024000" end="1026047"/>
-->    </partition>
-->    <partition number="2" type="primary" isActive="false" fileSystemType="">
-->     <device path="/dev/sda2" major="8" minor="2"/>
-->     <lba start="1026048" length="155275264" end="156301311"/>
-->    </partition>
-->   </disk>
-->   <volumeGroup version="2" name="vg_cleveland" capacityInBytes="79498838016" freeSpaceInBytes="0"/>
-->  </storage>
-->  <network>
-->   <interface label="eth0" nicId="xx:xx:xx:xx:xx:xx" status="up">
-->    <ip address="xxx.xxx.xxx.xxx" mask="xxx.xxx.xxx.xxx"/>
-->   </interface>
-->  </network>
-->  <operatingSystem>
-->   <hostname>xxx..xxx.xxx</hostname>
-->   <distribution name="rhel" version="6.3 (Santiago)"/>
-->   <kernel release="2.6.32-279.2.1.el6.x86_64" is64bit="true" SMP="false"/>
-->  </operatingSystem>
-->  <hardwareInfo>
-->   <firmwareType>bios</firmwareType>
-->   <cpuCount>4</cpuCount>
-->   <coresCount id="0">4</coresCount>
-->   <memoryInKb>3801088</memoryInKb>
-->  </hardwareInfo>
-->  <grub>
-->   <devicemap>
-->    <device biosNumber="0" path="/dev/sda"/>
-->   </devicemap>
-->  </grub>
--> </sysinfo>
-->

2012-08-08T12:49:16.506-05:00 [06060 info 'task-1'] Converter Agent SysinfoQuery obtained XML output
2012-08-08T12:49:16.506-05:00 [06060 info 'task-1'] Caching vmware-sysinfo XML output with key LiveLinuxComputer::GetSysinfoXml=d9373a5d11707ea981d6ba589b6bf8b286a8610
2012-08-08T12:49:16.508-05:00 [06060 warning 'task-1'] No volume for partition /dev/sda2

My best guess is that vmware-sysinfo thinks the ext4 partition with the LVM is an ext3 partition and that is causing problems for Converter to recognize the partiton.  The drive in this system is a single SATA drive connected to an onboard SATA port.

[ibodog]

Here is what fdisk has to say about the drive:

# fdisk -l /dev/sda

Disk /dev/sda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0001558d

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          64      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              64        9730    77637632   8e  Linux LVM

[Ivan Ivanov]

For some reason sysinfo recognized only one mounted volume (/boot). Is there a section (close to the sysinfo XML in the worker log) that is the actual log file of vmware-sysinfo execution? It should have log entries similar to "Storage.INFO"? This could shed some light on why sysinfo did not detect the / filesystem.

[ibodog]

No there are no such entries in that file with "storage" "INFO" or "Storage.INFO".

[EDIT]  Is there a logging level setting for Converter?

[Ivan Ivanov]

Can you switch the worker logging level to "verbose" and retry the conversion? Details on how to do it can be found at http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2008019. It is important to restart the worker service after editing the config file, however if you have any running conversion tasks they will be terminated.

[ibodog]

Looks like it can't make out the fstab lines for the main LVM partition:

2012-08-09T10:35:30.879-05:00 [07120 verbose 'Default'] cmdArgs[7]: cd /tmp/.vmware-sysinfo-yugrsrycrgxcbzix/; ./vmware-sysinfo-lin64.sh
2012-08-09T10:35:30.879-05:00 [07120 verbose 'SysCommandWin32'] Starting process: "C:\Program Files (x86)\VMware\VMware vCenter Converter Standalone\plink.exe" "-noprompt" "-stdin" "-P" "22" "-thumbprint" "85:4f:e3:e4:c7:8d:4e:28:3f:51:8b:39:17:d7:34:67" "[e-mail]" "cd /tmp/.vmware-sysinfo-yugrsrycrgxcbzix/; ./vmware-sysinfo-lin64.sh"
2012-08-09T10:35:32.859-05:00 [07120 info 'Default'] [Converter Agent SysinfoQuery] successfully exec'ed vmware-sysinfo; result: <?xml version="1.0" encoding="UTF-8"?>
--> <sysinfo>
-->  <storage>
-->   <volume mountPoint="/boot" blockSizeInBytes="1024" capacityInBlocks="495844" blocksFree="402778" blocksUsed="93066" devicePath="/dev/sda1" fileSystemType="ext4" isBootVolume="true" label=""/>
-->   <disk type="msdos" controllerType="scsi" bus="3" deviceId="0" capacityInSectors="156301488">
-->    <device path="/dev/sda" major="8" minor="0"/>
-->    <hwGeometry cylinders="9729" heads="255" sectors="63"/>
-->    <biosGeometry cylinders="9729" heads="255" sectors="63"/>
-->    <partition number="1" type="primary" isActive="true" fileSystemType="ext3">
-->     <device path="/dev/sda1" major="8" minor="1"/>
-->     <lba start="2048" length="1024000" end="1026047"/>
-->    </partition>
-->    <partition number="2" type="primary" isActive="false" fileSystemType="">
-->     <device path="/dev/sda2" major="8" minor="2"/>
-->     <lba start="1026048" length="155275264" end="156301311"/>
-->    </partition>
-->   </disk>
-->   <volumeGroup version="2" name="vg_cleveland" capacityInBytes="79498838016" freeSpaceInBytes="0"/>
-->  </storage>
-->  <network>
-->   <interface label="eth0" nicId="xx:xx:xx:xx:xx:xx" status="up">
-->    <ip address="xx.xx.xx.xx" mask="xx.xx.xx.xx"/>
-->   </interface>
-->  </network>
-->  <operatingSystem>
-->   <hostname>xxx.xxx.xxx</hostname>
-->   <distribution name="rhel" version="6.3 (Santiago)"/>
-->   <kernel release="2.6.32-279.2.1.el6.x86_64" is64bit="true" SMP="false"/>
-->  </operatingSystem>
-->  <hardwareInfo>
-->   <firmwareType>bios</firmwareType>
-->   <cpuCount>4</cpuCount>
-->   <coresCount id="0">4</coresCount>
-->   <memoryInKb>3801088</memoryInKb>
-->  </hardwareInfo>
-->  <grub>
-->   <devicemap>
-->    <device biosNumber="0" path="/dev/sda"/>
-->   </devicemap>
-->  </grub>
--> </sysinfo>
-->
2012-08-09T10:35:32.859-05:00 [07120 verbose 'Default'] Invoking C:\Program Files (x86)\VMware\VMware vCenter Converter Standalone\plink.exe with the following arguments:
2012-08-09T10:35:32.859-05:00 [07120 verbose 'Default'] cmdArgs[0]: -noprompt
2012-08-09T10:35:32.859-05:00 [07120 verbose 'Default'] cmdArgs[1]: -stdin
2012-08-09T10:35:32.859-05:00 [07120 verbose 'Default'] cmdArgs[2]: -P
2012-08-09T10:35:32.859-05:00 [07120 verbose 'Default'] cmdArgs[3]: 22
2012-08-09T10:35:32.859-05:00 [07120 verbose 'Default'] cmdArgs[4]: -thumbprint
2012-08-09T10:35:32.859-05:00 [07120 verbose 'Default'] cmdArgs[5]: 85:4f:e3:e4:c7:8d:4e:28:3f:51:8b:39:17:d7:34:67
2012-08-09T10:35:32.859-05:00 [07120 verbose 'Default'] cmdArgs[6]: [e-mail]
2012-08-09T10:35:32.859-05:00 [07120 verbose 'Default'] cmdArgs[7]: cat /tmp/.vmware-sysinfo-yugrsrycrgxcbzix/vmware-sysinfo.log
2012-08-09T10:35:32.859-05:00 [07120 verbose 'SysCommandWin32'] Starting process: "C:\Program Files (x86)\VMware\VMware vCenter Converter Standalone\plink.exe" "-noprompt" "-stdin" "-P" "22" "-thumbprint" "85:4f:e3:e4:c7:8d:4e:28:3f:51:8b:39:17:d7:34:67" "[e-mail]" "cat /tmp/.vmware-sysinfo-yugrsrycrgxcbzix/vmware-sysinfo.log"
2012-08-09T10:35:33.723-05:00 [07120 verbose 'Default'] Converter Agent SysinfoQuery grabbed log file; result: [2012-08-09 10:35:31,157 INFO storage ]: Probing for disks using parted
--> [2012-08-09 10:35:31,158 INFO storage ]: Parted returned device with path: /dev/sda, type: 1
--> [2012-08-09 10:35:31,170 INFO storage ]: Parted successfully probed device with path: /dev/sda, type: 1, disk type: msdos
--> [2012-08-09 10:35:31,170 INFO storage.disk ]: Creating SCSI disk with path: /dev/sda
--> [2012-08-09 10:35:31,295 INFO storage ]: LVM2 executables found
--> [2012-08-09 10:35:31,731 WARN storage.lvm ]: Failed to match lvs output line '  lv_root|vg_cleveland|73324822528B|-wi-ao--|1
--> '.
--> [2012-08-09 10:35:31,731 WARN storage.lvm ]: Failed to match lvs output line '  lv_swap|vg_cleveland|6174015488B|-wi-ao--|1
--> '.
--> [2012-08-09 10:35:31,732 INFO storage ]: Got mount table entry: fsname: /dev/mapper/vg_cleveland-lv_root, dir: /, type: ext4
--> [2012-08-09 10:35:31,732 INFO storage ]: Looking for device with major 253 and minor 0
--> [2012-08-09 10:35:31,732 INFO storage ]: Got mount table entry: fsname: proc, dir: /proc, type: proc
--> [2012-08-09 10:35:31,732 INFO storage ]: Got mount table entry: fsname: sysfs, dir: /sys, type: sysfs
--> [2012-08-09 10:35:31,732 INFO storage ]: Got mount table entry: fsname: devpts, dir: /dev/pts, type: devpts
--> [2012-08-09 10:35:31,732 INFO storage ]: Got mount table entry: fsname: tmpfs, dir: /dev/shm, type: tmpfs
--> [2012-08-09 10:35:31,732 INFO storage ]: Got mount table entry: fsname: /dev/sda1, dir: /boot, type: ext4
--> [2012-08-09 10:35:31,732 INFO storage ]: Looking for device with major 8 and minor 1
--> [2012-08-09 10:35:31,732 INFO storage ]: Found device /dev/sda1
--> [2012-08-09 10:35:31,732 WARN storage.volume ]: Creating volume with mountPoint: /boot, device path: /dev/sda1, fsType: ext4, _blockSizeInBytes: 1024, _capacityInBlocks: 495844, _blocksFree: 402778
--> [2012-08-09 10:35:31,732 INFO storage.volume ]: Device /dev/sda1 is not a FAT volume
--> [2012-08-09 10:35:31,732 WARN storage.volume ]: Unrecognized file system on device /dev/sda1, label will not be preserved
--> [2012-08-09 10:35:31,732 INFO storage ]: Got mount table entry: fsname: none, dir: /proc/sys/fs/binfmt_misc, type: binfmt_misc
--> [2012-08-09 10:35:31,732 INFO storage ]: Got mount table entry: fsname: sunrpc, dir: /var/lib/nfs/rpc_pipefs, type: rpc_pipefs
--> [2012-08-09 10:35:31,732 INFO storage ]: Getting stats for swap device /dev/dm-1
--> [2012-08-09 10:35:31,732 INFO storage ]: Swap device /dev/dm-1 has block device number (major,minor): (253,1)
--> [2012-08-09 10:35:31,734 WARN network ]: Ignoring interface 'lo' because it is not an ethernet interface
--> [2012-08-09 10:35:31,734 INFO distribution ]: /etc/redhat-release file line: 'Red Hat Enterprise Linux Server release 6.3 (Santiago)'
--> [2012-08-09 10:35:31,751 INFO hardwareInfo ]: /proc/sys/kernel/bootloader_type has valie: '113'
--> [2012-08-09 10:35:31,752 INFO grub ]: Trying to create device map file /tmp/vmware-sysinfo-grub-device.map
--> [2012-08-09 10:35:31,752 INFO grub ]: Running command: grub --batch --no-floppy --device-map=/tmp/vmware-sysinfo-grub-device.map > /dev/null 2>&1
--> [2012-08-09 10:35:32,325 INFO sysinfo ]: <?xml version="1.0" encoding="UTF-8"?>

I think the OS is a relatively "vanilla" install onto the 80GB disk so I'm surprised to see this.  How would this be resolved?

[Ivan Ivanov]

Yep, the problem is in lvs output. vmware-sysinfo is running

# lvm lvs -o lv_name,vg_name,lv_size,lv_attr,stripes --units b --noheadings --unbuffered --separator \| vg_cleveland

in order to extract logical volumes information on volume group 'vg_cleveland'. The log says the result of this command is

  lv_root|vg_cleveland|73324822528B|-wi-ao--|1
  lv_swap|vg_cleveland|6174015488B|-wi-ao--|1

and then vmware-sysinfo is failing to parse these lines, so it cannot recognize the LVM volumes and that's why later it cannot find a match for the root filesystem mountpoint, which leads to eventually filtering it out of the result.

The problem in lvs output is in the fourth column (with LVM volume properties). vmware-sysinfo expects only 6 characters, however in the current lines ther are 8 of them. I am not sure, but is it possible to have a special lvm implementation by Oracle that generates two additional characters in that column and in this manner to cause the lvs output parser to fail.

For a workaround - cannot think of anything easy right now. On Converter side, it includes rebuilding vmware-sysinfo executable. On LVM side - it either includes rebuilding/installing an LVM package that generates the output in a form recognized by Converter (I have no idea what caused this difference - whether an Oracle-specific version or the lvm binary on the source is much newer than Converter and there is a change in the output of lvs command. The ways of handling this I can think of right now are

1. to install standard/older LVM package on the source that gnerates the output in the expected format (cannot tell what version since it is not clear what is causing this issue)
2. to rename lvm binary to something else and create a wrapper script around it called lvm that "fixes" the result of the original lvs command before returning it to vmware-sysinfo (e. g. using sed).
3. to rebuild vmware-sysinfo binary (which is opensource) with fixing the regular expression to recognize this output.

However, keep in mind that Oracle Linuxis not officially supported or tested so no matter whether you handle this problem or not, there are chances that the target VM won't boot up because of failed reconfiguration. If this happens you should boot from a live CD and try to manually fix the target VM in order to make it bootable (update fstab, reinstall GRUB, rebuild initrd, etc.).

HTH,

Ivan

[ibodog]

The Oracle Linux 6 install is using lvm2-2.02.95-10.el6.x86_64 (Oracle Linux Server release 6.3).

I've got a CentOS 6 install that is using lvm2-2.02.87-6.el6.x86_64  (CentOS release 6.2 (Final)).  Converter does show the LVM partition  correctly there.  I ran a "yum update" on this system and after rebooting it is using lvm2-2.02.95-10.el6.x86_64 (CentOS release 6.3 (Final)).  So I ran Converter on that system after the update.  This CentOS 6 system now fails to have the LVM partition recognized the same way as the Oracle Linux 6 system.

I'd guess that RHEL 6r3 would have the same problem.

[Ivan Ivanov]

Yeah, it seems the output has changed in the newer versions so we need to update Converter accordingly... but still the workaround with the older LVM version on the source (how older?) should work...

[ibodog]

It looks like lvm2 2.02.87 was the previous version.

So to downgrade I did this as root:

yum remove lvm2

yum remove lvm2-libs

then

yum install lvm2-2.02.87

Now Converter 5 does recognize the LVM partition on the Oracle Linux 6 system!

Thanks!!

[eilatuno]

Yes i confirm you need to downgrade the package lvm2 and lvm2 libs to prevoius version.

But the next problem, is the operation failed at 99% when VMwareconverter try to reconfigure the new VM because REDHAT6 is not supported by converter5.

I solved the problem with this small procedure:

1-Unflag the reconfigure option in Vmware Converter import wizard

2-After the conversion the vm is non bootable,

     -boot the vm with a rescue-cd or a knoppix,

     -chroot the system

     -edit your fstab to match new partition schema(for example: if you physical server was an HP ,probably the /boot partition is /dev/c0d0p1, change        this to the proper device probably /dev/sda1)

     -recreate the boot image with mkinitrd tool

     -edit inittab to 1 (if you haven't made modification at next boot the system try to start all services and probably fail)

     -shutdown

3-Now your system it's half ready to boot,one of my big problem after the conversion ,was the unable to boot because the boot partition lost grub configuration. to fix this start the vm with supergrub boot disk (http://www.supergrubdisk.org/) and boot the system, now reinstall grub on boot partion with grub-install /dev/sda1 (probably you need to do a --recheck before install) and reboot without the supergrub iso.

4-The server is now booting fine,  install the vmware tools and you complete the P2V!

I hope this help anyone.

Sorry for the bad english!

[pbritoferreira]

THANKS A LOT, with capital letters, eliatuno, that helped me a LOT! I was almost deciding to recreate the machine from scratch and reconfigure and etc.

Just so if anyone needs detailed info, this is exactly what I did using an Ubuntu 12.04 live to rescue:

sudo su
export http_proxy=http://@10.0.0.111:80 ### proxy info
echo 'Acquire::http::Pipeline-Depth "0";' >> /etc/apt/apt.conf.d/workaround.conf ### required to work with my proxy
apt-get install -y vim lvm2
modprobe dm-mod
vgchange -ay
mount /dev/YOUR_VG/YOUR_ROOT_LV /mnt/
mount -o bind /proc /mnt/proc
mount -o bind /sys /mnt/sys
mount -o bind /dev /mnt/dev
mount /dev/sda1 /mnt/boot
chroot /mnt

mkinitrd --preload lvm2 initrd-2.6.32-279.5.2.el6.x86_64.img 2.6.32-279.5.2.el6.x86_64 --force ### Recreate the initrd with lvm support (this was the tipping point for me, lvm support)

vim /etc/grub.conf ### point the initrd directive to your new initrd

grub-install hd0

exit

reboot

These were the exact commands I typed, hope this helps someone =)

[larvel]

Simple workaround is to swap the lvm binary: (Installing the rpm was not a solution for me)

Fetch working lvm:

wget ftp://ftp.ntua.gr/pub/linux/scientificlinux/6.2/x86_64/os/Packages/lvm2-2.02.87-6.el6.x86_64.rpm

Unpack rpm

rpm2cpio lvm2-2.02.87-6.el6.x86_64.rpm | cpio -idmv

Temporarily rename original lvm:

mv /sbin/lvm /sbin/lvm.original

The tell the lvm-files apart, simply make a symbolic link:

ln -s path-to new-lvm /sbin/lvm

..and of course  remember to swap the lvm`s back.

After the cloning is done, simply boot the vm in rescue mode, enter chroot and perform

dracut --mdadmconf --force /boot/initramfs-2.6.32-220.13.1.el6.x86_64.img 2.6.32-220.13.1.el6.x86_64

check grub.conf to find the proper kernel version.

Cheers!